Process of notebook:
- Problem Definition
- Data
- Evaluate
- Features
1. Problem Definition¶
How can I predict my body weight (in kgs), in addition to my waist circumference (in cm), just by using the calories I've consumed, in addition to my macronutrients (in grams) and it's percentage in calories?
2. Data¶
We're working with semi-structured excel sheet data. The data I recorded was both weight and waist measurement, in the morning and in the night. But I've slowly transitioned to only measuring in the morning instead. And I didn't track my calories and macros until a little later into my excel recordings. So there will be a number of data cleaning.
We'll be working with date-time data, as the weight and waist data is associated to its date-time, in addition to the calories and macro details.
There is no established train or test dataset. So we'll split the roughly 3800 cels into these 2 sets of dataframes.
3. Evaluation¶
We will evaluate by how well the ML program could predict my weight, only by the calories and macro data, in addition to using my weight and waist from the previous days as reference for its prediction.
4. Features¶
We will only have 7 different features to given, in order to predict my weight and waist. But we will also need to factor in my weight and waist from the previous day/s as well, as my current body weight, also has an affect on the weight I either gain or lose on the following days.
We can form 2 extra features to help us. The positive and negative difference between the weight and waist results from the previous day, to current day.
In [1]:
# Timestamp - to make recording on when this project was first made
import datetime
print(f'Start of notebook run: {datetime.datetime.now()}')
Start of notebook run: 2025-04-15 15:32:05.169992
In [2]:
# import analysis tools
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
In [3]:
# Import excel file
!pip install openpyxl
data = pd.read_excel('Weight watching data.xlsx')
data.head()
Requirement already satisfied: openpyxl in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python39\site-packages (3.1.5) Requirement already satisfied: et-xmlfile in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python39\site-packages (from openpyxl) (2.0.0)
WARNING: You are using pip version 22.0.4; however, version 25.0.1 is available. You should consider upgrading via the 'C:\Users\deanc\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.9_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip' command.
Out[3]:
| Date | Notes | Weight (kg) | Unnamed: 3 | Waist (cm) | Unnamed: 5 | BMI | Unnamed: 7 | Unnamed: 8 | Unnamed: 9 | ... | Unnamed: 58 | Unnamed: 59 | Unnamed: 60 | Unnamed: 61 | Unnamed: 62 | Unnamed: 63 | Unnamed: 64 | Unnamed: 65 | Unnamed: 66 | Unnamed: 67 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Start - Aug 20th 2019 | NaN | 53.7 | NaN | 66.0 | NaN | 17.736821 | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | NaN | NaN | 53.6 | NaN | 64.5 | NaN | 17.703792 | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | NaN | NaN | 53.7 | NaN | 64.0 | NaN | 17.736821 | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 3 | NaN | NaN | 53.2 | NaN | 68.2 | NaN | 17.571674 | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | NaN | NaN | 52.9 | NaN | 66.2 | NaN | 17.472586 | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
5 rows × 68 columns
In [4]:
data.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 30637 entries, 0 to 30636 Data columns (total 68 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 30172 non-null object 1 Notes 15 non-null object 2 Weight (kg) 3853 non-null float64 3 Unnamed: 3 2 non-null object 4 Waist (cm) 3853 non-null float64 5 Unnamed: 5 0 non-null float64 6 BMI 3853 non-null float64 7 Unnamed: 7 0 non-null float64 8 Unnamed: 8 74 non-null object 9 Unnamed: 9 1 non-null object 10 Unnamed: 10 8490 non-null object 11 Unnamed: 11 0 non-null float64 12 Unnamed: 12 1959 non-null object 13 Unnamed: 13 0 non-null float64 14 Unnamed: 14 3916 non-null object 15 Unnamed: 15 0 non-null float64 16 Fast length (hours) 4156 non-null object 17 Unnamed: 17 1 non-null object 18 Fast avg. 7 (hours) 4150 non-null object 19 Unnamed: 19 0 non-null float64 20 Height 1690 non-null object 21 Unnamed: 21 0 non-null float64 22 Protein (g) 1690 non-null object 23 Unnamed: 23 0 non-null float64 24 Fat (g) 1689 non-null object 25 Unnamed: 25 0 non-null float64 26 Carbohydrate % 1689 non-null object 27 Unnamed: 27 0 non-null float64 28 Protein % 1689 non-null object 29 Unnamed: 29 0 non-null float64 30 Fat % 1689 non-null object 31 Unnamed: 31 0 non-null float64 32 Macro score 1689 non-null object 33 Unnamed: 33 0 non-null float64 34 Score*Log. Kcal 1689 non-null object 35 Unnamed: 35 0 non-null float64 36 Weight (kg).1 3916 non-null object 37 Unnamed: 37 0 non-null float64 38 Waist (cm).1 3916 non-null object 39 Unnamed: 39 0 non-null float64 40 Unnamed: 40 4522 non-null object 41 Unnamed: 41 0 non-null float64 42 Unnamed: 42 2013 non-null object 43 Unnamed: 43 0 non-null float64 44 Unnamed: 44 221 non-null object 45 Unnamed: 45 0 non-null float64 46 Unnamed: 46 0 non-null float64 47 Unnamed: 47 0 non-null float64 48 Unnamed: 48 0 non-null float64 49 Unnamed: 49 0 non-null float64 50 Unnamed: 50 0 non-null float64 51 Unnamed: 51 0 non-null float64 52 Unnamed: 52 0 non-null float64 53 Unnamed: 53 0 non-null float64 54 Unnamed: 54 0 non-null float64 55 Unnamed: 55 0 non-null float64 56 Unnamed: 56 0 non-null float64 57 Unnamed: 57 0 non-null float64 58 Unnamed: 58 0 non-null float64 59 Unnamed: 59 0 non-null float64 60 Unnamed: 60 0 non-null float64 61 Unnamed: 61 0 non-null float64 62 Unnamed: 62 0 non-null float64 63 Unnamed: 63 0 non-null float64 64 Unnamed: 64 0 non-null float64 65 Unnamed: 65 0 non-null float64 66 Unnamed: 66 0 non-null float64 67 Unnamed: 67 1 non-null object dtypes: float64(43), object(25) memory usage: 15.9+ MB
Unfortunately, there's a lot of NaN values in my dataset. Fortunately, I'm familiar with what my data set is.
If I've filled in any type of value in my excel sheet, then its a valid point of my data. Any data with missing values, will not be counted, as I have not entered any value in the cels.
The columns with few to no values, are either used for communication purposes, or column separation to make my data input/viewing more easier on the eyes respectfully. Therefore can be removed, due to no particular use to our goals of this project.
In [5]:
# Remove columns with 1000 or fewer variables within it
df = data.loc[:, data.count() > 1000] # .loc = locating something from our data, representing = [row:column]
# aka counting 1000 or more values in our column, and locating them into our new variable, df
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 30637 entries, 0 to 30636 Data columns (total 21 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 30172 non-null object 1 Weight (kg) 3853 non-null float64 2 Waist (cm) 3853 non-null float64 3 BMI 3853 non-null float64 4 Unnamed: 10 8490 non-null object 5 Unnamed: 12 1959 non-null object 6 Unnamed: 14 3916 non-null object 7 Fast length (hours) 4156 non-null object 8 Fast avg. 7 (hours) 4150 non-null object 9 Height 1690 non-null object 10 Protein (g) 1690 non-null object 11 Fat (g) 1689 non-null object 12 Carbohydrate % 1689 non-null object 13 Protein % 1689 non-null object 14 Fat % 1689 non-null object 15 Macro score 1689 non-null object 16 Score*Log. Kcal 1689 non-null object 17 Weight (kg).1 3916 non-null object 18 Waist (cm).1 3916 non-null object 19 Unnamed: 40 4522 non-null object 20 Unnamed: 42 2013 non-null object dtypes: float64(3), object(18) memory usage: 4.9+ MB
So now we have a range of different numbers, still with many NaN. How do we figure out what to throw or keep?
We will keep the column's data, that we'll be utilizing in our project, that has the least amount of inputted data. For our example, 1,689 is currently the lowest number of data we have that we'll be using. Which is namely our macro data. We do have a weird column named height though with 1,690 values.
Let's check out the dataframe and have a deeper dive what it displays.
In [6]:
df.head()
Out[6]:
| Date | Weight (kg) | Waist (cm) | BMI | Unnamed: 10 | Unnamed: 12 | Unnamed: 14 | Fast length (hours) | Fast avg. 7 (hours) | Height | ... | Fat (g) | Carbohydrate % | Protein % | Fat % | Macro score | Score*Log. Kcal | Weight (kg).1 | Waist (cm).1 | Unnamed: 40 | Unnamed: 42 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | Start - Aug 20th 2019 | 53.7 | 66.0 | 17.736821 | NaN | NaN | NaN | 13 | 20 | 1.74 | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | NaN | 53.6 | 64.5 | 17.703792 | NaN | NaN | NaN | 26 | 21.142857 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | NaN | 53.7 | 64.0 | 17.736821 | NaN | NaN | NaN | 21 | 19.428571 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 3 | NaN | 53.2 | 68.2 | 17.571674 | NaN | NaN | NaN | 15 | 19.714286 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | NaN | 52.9 | 66.2 | 17.472586 | NaN | NaN | NaN | 32 | 20.428571 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
5 rows × 21 columns
With Height, it's likely one singular datapoint to hold the variable of my height, to use to calculate and do some feature engineering in my excel sheet. For example, my BMI data.
But protein has 1,690 values. So I wonder what extra variable is in it?
In [7]:
# Separate numeric and string values
numeric_values = pd.to_numeric(df['Protein (g)'], errors='coerce') # convert numeric values, and leave NaN for any non-numeric variables
strings = df['Protein (g)'][df['Protein (g)'].apply(lambda x: isinstance(x, str))] # filters for rows that are strings
# Sort each group independently
numeric_values_sorted = sorted(numeric_values.dropna(), reverse=True)
strings_sorted = sorted(strings)
# Combine them if needed
sorted_values = numeric_values_sorted + strings_sorted
print(sorted_values)
[374.4, 356.4, 333.6, 317.7, 313.2, 302.1, 294.7, 294.7, 279.0, 277.9, 276.6, 272.8, 270.5, 268.8, 268.6, 268.5, 265.2, 263.9, 263.9, 263.8, 263.3, 263.0, 262.4, 260.4, 259.5, 258.2, 257.9, 256.4, 254.5, 253.5, 253.3, 252.6, 252.2, 252.2, 251.1, 251.1, 250.6, 250.5, 249.5, 249.3, 247.9, 247.4, 246.0, 243.9, 243.6, 242.5, 241.8, 241.5, 239.4, 236.0, 235.9, 235.6, 235.3, 234.7, 234.5, 234.2, 233.3, 232.8, 230.9, 230.4, 229.9, 229.7, 229.5, 228.4, 227.7, 227.5, 227.1, 225.6, 225.6, 225.5, 224.9, 224.4, 224.1, 223.9, 223.6, 222.7, 222.6, 222.1, 221.4, 220.8, 220.6, 220.5, 220.1, 219.7, 219.2, 218.7, 218.4, 218.2, 217.7, 216.5, 216.4, 216.2, 216.1, 215.9, 215.9, 215.6, 215.4, 214.9, 214.6, 214.5, 214.2, 214.1, 214.0, 213.9, 213.0, 212.6, 212.5, 212.3, 212.3, 211.9, 211.7, 211.5, 211.3, 210.8, 210.7, 210.6, 210.3, 209.6, 209.4, 209.3, 208.5, 207.7, 207.6, 207.5, 207.2, 207.2, 207.1, 207.1, 207.0, 206.9, 206.8, 206.2, 205.9, 204.9, 204.4, 204.1, 203.9, 203.8, 203.4, 203.4, 202.8, 202.5, 202.4, 202.3, 201.5, 201.5, 201.1, 200.8, 200.5, 200.3, 200.1, 200.0, 199.9, 199.3, 198.1, 198.0, 197.9, 197.3, 197.3, 196.8, 196.8, 196.8, 196.2, 196.2, 196.2, 196.1, 195.6, 195.3, 195.1, 195.0, 194.9, 194.8, 194.6, 194.1, 194.0, 193.2, 193.0, 193.0, 192.1, 192.0, 191.8, 191.8, 191.7, 191.7, 191.6, 191.4, 191.4, 191.4, 191.1, 190.2, 190.1, 190.0, 190.0, 189.6, 189.3, 189.2, 189.1, 188.7, 188.5, 188.5, 188.4, 188.3, 188.2, 188.1, 187.7, 187.7, 187.4, 187.2, 187.0, 186.8, 186.3, 185.9, 185.7, 185.3, 185.1, 184.8, 184.7, 184.7, 184.5, 184.5, 184.3, 184.3, 184.3, 184.0, 184.0, 184.0, 183.6, 183.6, 183.4, 183.4, 183.4, 183.3, 183.2, 183.1, 183.0, 182.7, 182.6, 182.3, 182.1, 182.0, 181.9, 181.2, 180.6, 180.4, 180.2, 179.2, 179.2, 179.1, 179.0, 179.0, 178.8, 178.8, 178.7, 178.3, 178.0, 178.0, 177.7, 177.4, 177.3, 177.0, 177.0, 177.0, 176.8, 176.8, 176.8, 176.5, 176.4, 176.2, 175.9, 175.8, 175.5, 175.4, 175.3, 175.2, 175.0, 175.0, 175.0, 174.9, 174.7, 174.5, 174.3, 174.2, 174.1, 173.8, 173.6, 173.5, 173.5, 173.2, 172.8, 172.7, 172.2, 172.0, 171.9, 171.9, 171.4, 171.2, 171.1, 170.8, 170.6, 170.3, 170.0, 169.9, 169.9, 169.6, 169.4, 169.2, 169.2, 168.7, 168.5, 168.4, 168.3, 168.1, 167.6, 167.6, 167.4, 167.4, 167.4, 167.2, 166.8, 166.4, 166.4, 166.3, 166.1, 166.0, 165.9, 165.5, 165.4, 165.4, 165.3, 165.1, 165.1, 165.0, 164.9, 164.8, 164.8, 164.8, 164.3, 164.1, 163.9, 163.6, 163.4, 163.4, 163.4, 163.3, 163.2, 163.0, 163.0, 162.9, 162.5, 162.5, 162.3, 162.0, 161.5, 161.5, 161.5, 161.0, 160.5, 159.8, 159.7, 159.7, 159.4, 159.3, 159.2, 158.7, 158.7, 158.5, 158.5, 158.5, 158.3, 157.9, 157.6, 157.6, 157.0, 156.6, 156.3, 156.3, 156.2, 156.0, 156.0, 155.7, 155.5, 155.2, 155.1, 155.1, 154.9, 154.8, 154.7, 154.4, 154.1, 154.0, 153.9, 153.4, 152.6, 152.0, 151.7, 151.5, 151.5, 151.5, 151.2, 151.0, 151.0, 150.5, 150.4, 150.1, 149.9, 149.7, 149.2, 149.1, 148.9, 148.9, 148.6, 148.5, 148.3, 148.3, 148.3, 148.2, 148.0, 147.9, 147.8, 147.7, 147.3, 146.8, 146.8, 146.7, 146.7, 146.6, 146.5, 146.5, 146.2, 145.7, 145.5, 144.4, 144.1, 144.0, 143.7, 143.7, 143.7, 143.6, 143.3, 143.0, 142.9, 142.9, 142.6, 142.5, 142.2, 142.2, 141.9, 141.8, 141.7, 141.6, 141.3, 141.1, 140.9, 140.8, 140.4, 140.2, 140.0, 139.9, 139.5, 139.5, 139.4, 139.2, 138.9, 138.8, 138.7, 138.6, 138.4, 138.3, 137.8, 137.7, 137.2, 136.9, 136.4, 136.3, 136.1, 136.0, 135.6, 135.5, 135.0, 134.7, 134.5, 134.4, 134.2, 134.1, 134.1, 133.9, 133.6, 132.8, 132.8, 132.7, 132.7, 132.6, 132.5, 132.5, 132.1, 132.1, 131.7, 131.5, 131.0, 130.9, 130.5, 130.5, 130.4, 130.3, 130.3, 130.2, 129.6, 129.6, 129.3, 129.2, 129.0, 128.8, 128.7, 128.7, 128.4, 128.4, 128.4, 128.2, 128.1, 128.1, 128.1, 128.1, 127.8, 127.5, 127.5, 127.5, 127.4, 127.3, 127.1, 126.9, 126.7, 126.4, 126.0, 126.0, 125.8, 125.4, 125.3, 125.1, 125.0, 125.0, 124.9, 124.7, 124.2, 124.2, 124.1, 123.9, 123.1, 123.0, 123.0, 123.0, 122.6, 122.6, 122.5, 122.5, 122.4, 122.4, 122.1, 122.1, 121.1, 121.0, 120.9, 120.6, 120.3, 120.2, 120.0, 119.8, 119.3, 119.1, 119.1, 119.0, 118.8, 118.8, 118.7, 118.6, 118.5, 118.4, 118.1, 118.0, 117.9, 117.8, 117.5, 117.5, 117.5, 117.1, 117.1, 116.8, 116.7, 116.7, 116.6, 116.4, 116.3, 116.1, 116.1, 116.0, 116.0, 115.9, 115.9, 115.5, 115.5, 115.5, 115.2, 115.2, 114.9, 114.2, 114.2, 114.1, 114.1, 114.0, 113.7, 113.7, 113.7, 113.7, 113.3, 113.1, 113.1, 113.0, 112.9, 112.9, 112.8, 112.5, 112.3, 111.8, 111.7, 111.5, 111.0, 110.9, 110.7, 110.4, 110.3, 110.1, 110.0, 109.9, 109.9, 109.6, 109.6, 109.6, 109.5, 109.4, 109.3, 109.3, 109.2, 109.2, 109.1, 109.1, 109.1, 109.0, 109.0, 108.9, 108.7, 108.7, 108.7, 108.5, 108.0, 107.9, 107.6, 107.5, 107.5, 107.5, 107.2, 107.1, 107.1, 107.0, 107.0, 106.8, 106.8, 106.6, 106.4, 106.3, 106.2, 106.1, 106.0, 105.9, 105.6, 105.5, 105.4, 105.4, 105.4, 105.3, 105.2, 105.2, 105.1, 105.0, 105.0, 105.0, 104.6, 104.6, 104.5, 104.4, 104.2, 104.1, 104.0, 104.0, 104.0, 104.0, 103.7, 103.7, 103.6, 103.3, 103.0, 102.7, 102.7, 102.5, 102.5, 102.4, 102.4, 102.3, 102.1, 102.0, 102.0, 101.9, 101.8, 101.7, 101.7, 101.7, 101.6, 101.5, 101.5, 101.5, 101.4, 101.4, 101.4, 101.3, 101.3, 101.2, 101.2, 101.2, 101.1, 100.9, 100.8, 100.6, 100.5, 100.4, 100.4, 100.2, 100.1, 100.0, 100.0, 100.0, 99.9, 99.7, 99.7, 99.5, 99.5, 99.4, 99.4, 99.3, 99.0, 99.0, 99.0, 98.9, 98.6, 98.5, 98.4, 98.4, 98.3, 98.2, 98.2, 98.1, 98.0, 97.9, 97.8, 97.6, 97.3, 97.2, 97.2, 97.2, 97.2, 97.1, 97.0, 96.9, 96.8, 96.7, 96.7, 96.5, 96.4, 96.3, 96.1, 96.0, 96.0, 95.9, 95.8, 95.8, 95.8, 95.7, 95.5, 95.4, 95.4, 95.3, 95.2, 95.1, 95.1, 95.1, 94.8, 94.8, 94.5, 94.5, 94.4, 94.4, 94.4, 94.3, 94.3, 94.2, 94.1, 94.1, 93.9, 93.8, 93.8, 93.8, 93.7, 93.7, 93.6, 93.4, 93.4, 93.3, 93.1, 93.1, 93.0, 93.0, 92.9, 92.9, 92.8, 92.7, 92.7, 92.7, 92.6, 92.6, 92.6, 92.6, 92.5, 92.5, 92.5, 92.5, 92.4, 92.3, 92.3, 92.2, 92.2, 92.1, 92.0, 92.0, 91.9, 91.9, 91.8, 91.7, 91.6, 91.6, 91.6, 91.5, 91.5, 91.4, 91.3, 91.1, 91.1, 91.0, 91.0, 91.0, 91.0, 90.8, 90.7, 90.7, 90.7, 90.6, 90.5, 90.2, 90.2, 90.1, 90.1, 90.0, 89.7, 89.7, 89.5, 89.5, 89.4, 89.0, 89.0, 88.9, 88.9, 88.7, 88.5, 88.5, 88.3, 88.3, 88.3, 88.1, 88.1, 87.9, 87.9, 87.8, 87.7, 87.6, 87.5, 87.5, 87.5, 87.5, 87.3, 87.0, 86.9, 86.8, 86.8, 86.7, 86.7, 86.6, 86.5, 86.4, 86.4, 86.4, 86.4, 86.4, 86.3, 86.3, 86.2, 86.2, 86.2, 86.1, 86.1, 86.1, 86.0, 85.9, 85.9, 85.9, 85.6, 85.4, 85.4, 85.3, 85.2, 85.2, 85.1, 85.0, 85.0, 84.9, 84.9, 84.7, 84.7, 84.6, 84.6, 84.4, 84.4, 84.3, 84.3, 84.3, 84.2, 84.1, 84.1, 84.1, 84.0, 83.9, 83.9, 83.9, 83.9, 83.8, 83.8, 83.6, 83.6, 83.2, 83.2, 83.1, 83.1, 83.1, 83.0, 83.0, 83.0, 82.9, 82.9, 82.9, 82.7, 82.7, 82.7, 82.5, 82.5, 82.5, 82.4, 82.1, 82.0, 82.0, 82.0, 82.0, 81.9, 81.8, 81.8, 81.6, 81.5, 81.5, 81.5, 81.4, 81.4, 81.2, 81.2, 81.2, 81.2, 81.1, 81.1, 81.1, 80.9, 80.8, 80.7, 80.7, 80.6, 80.5, 80.4, 80.4, 80.2, 80.1, 80.0, 80.0, 80.0, 79.9, 79.3, 79.3, 79.3, 79.3, 79.3, 79.0, 79.0, 79.0, 79.0, 78.8, 78.8, 78.6, 78.5, 78.4, 78.4, 78.3, 78.3, 78.2, 78.1, 78.1, 78.0, 77.8, 77.7, 77.6, 77.6, 77.4, 77.4, 77.4, 77.4, 77.3, 77.2, 77.2, 77.1, 77.1, 77.0, 77.0, 76.9, 76.9, 76.8, 76.7, 76.7, 76.6, 76.6, 76.6, 76.6, 76.6, 76.5, 76.5, 76.5, 76.2, 76.0, 76.0, 76.0, 76.0, 75.7, 75.5, 75.4, 75.2, 75.2, 75.1, 75.0, 75.0, 74.9, 74.8, 74.7, 74.7, 74.7, 74.3, 74.3, 74.2, 74.1, 74.1, 74.1, 74.0, 73.9, 73.9, 73.8, 73.7, 73.7, 73.6, 73.6, 73.5, 73.5, 73.5, 73.3, 73.3, 73.3, 73.2, 73.2, 73.2, 73.1, 73.1, 73.0, 73.0, 72.9, 72.9, 72.8, 72.8, 72.8, 72.8, 72.6, 72.5, 72.5, 72.5, 72.5, 72.4, 72.4, 72.4, 72.3, 72.3, 72.3, 72.2, 72.2, 72.0, 72.0, 72.0, 71.9, 71.9, 71.8, 71.8, 71.8, 71.7, 71.5, 71.4, 71.4, 71.3, 71.3, 71.1, 71.1, 71.0, 71.0, 70.9, 70.8, 70.7, 70.7, 70.7, 70.7, 70.6, 70.5, 70.5, 70.4, 70.4, 70.3, 70.3, 70.2, 70.2, 70.1, 70.0, 70.0, 69.9, 69.8, 69.6, 69.6, 69.5, 69.3, 69.2, 69.1, 69.1, 69.0, 69.0, 68.9, 68.8, 68.7, 68.7, 68.7, 68.5, 68.5, 68.2, 68.1, 68.0, 68.0, 68.0, 68.0, 67.9, 67.9, 67.8, 67.7, 67.6, 67.5, 67.5, 67.3, 67.3, 66.9, 66.9, 66.8, 66.8, 66.7, 66.6, 66.5, 66.4, 66.3, 66.2, 66.2, 66.1, 66.1, 66.0, 66.0, 65.9, 65.9, 65.9, 65.7, 65.7, 65.6, 65.6, 65.4, 65.3, 65.1, 65.1, 65.0, 65.0, 65.0, 64.9, 64.9, 64.8, 64.7, 64.7, 64.5, 64.5, 64.2, 64.2, 64.2, 64.0, 64.0, 63.8, 63.7, 63.7, 63.6, 63.6, 63.6, 63.6, 63.5, 63.4, 63.4, 63.3, 63.2, 63.0, 62.9, 62.9, 62.8, 62.8, 62.4, 62.3, 62.3, 62.3, 62.2, 62.1, 62.0, 62.0, 62.0, 62.0, 61.6, 61.5, 61.5, 61.5, 61.5, 61.5, 61.4, 61.4, 61.4, 61.2, 61.2, 61.1, 61.0, 61.0, 61.0, 60.9, 60.9, 60.9, 60.7, 60.6, 60.5, 60.5, 60.3, 60.2, 60.0, 60.0, 60.0, 60.0, 60.0, 59.8, 59.8, 59.6, 59.6, 59.4, 59.4, 59.2, 59.1, 59.0, 59.0, 58.5, 58.5, 58.5, 58.4, 58.1, 58.1, 58.1, 58.1, 58.0, 58.0, 58.0, 57.8, 57.8, 57.8, 57.8, 57.6, 57.5, 57.5, 57.5, 57.4, 57.3, 57.2, 57.2, 57.2, 57.2, 57.1, 57.1, 57.0, 57.0, 57.0, 57.0, 56.9, 56.8, 56.7, 56.6, 56.5, 56.5, 56.4, 56.4, 56.3, 56.3, 56.2, 56.1, 56.1, 56.0, 56.0, 55.7, 55.6, 55.5, 55.5, 55.5, 55.4, 55.4, 55.3, 55.3, 55.3, 55.0, 55.0, 55.0, 54.9, 54.8, 54.6, 54.6, 54.6, 54.5, 54.3, 54.2, 54.0, 53.9, 53.9, 53.5, 53.5, 53.5, 53.4, 53.3, 53.3, 53.2, 53.2, 53.1, 53.1, 52.9, 52.8, 52.8, 52.5, 52.5, 52.4, 52.4, 52.3, 52.3, 52.3, 52.2, 52.2, 52.2, 52.1, 52.0, 52.0, 51.9, 51.8, 51.8, 51.7, 51.6, 51.5, 51.5, 51.4, 51.4, 51.4, 51.3, 51.2, 51.2, 51.1, 51.1, 51.0, 51.0, 51.0, 50.9, 50.9, 50.8, 50.6, 50.5, 50.5, 50.3, 50.2, 50.1, 50.1, 50.0, 49.9, 49.7, 49.6, 49.4, 49.4, 49.1, 49.0, 49.0, 48.9, 48.9, 48.9, 48.9, 48.6, 48.5, 48.5, 48.4, 48.3, 48.3, 48.2, 48.0, 48.0, 48.0, 47.8, 47.8, 47.7, 47.6, 47.6, 47.5, 47.5, 47.2, 47.2, 47.2, 47.2, 47.0, 47.0, 47.0, 47.0, 47.0, 46.8, 46.8, 46.7, 46.5, 46.2, 46.2, 46.1, 46.1, 46.0, 46.0, 46.0, 46.0, 45.8, 45.7, 45.7, 45.6, 45.6, 45.5, 45.4, 45.4, 45.4, 45.2, 45.1, 45.0, 45.0, 45.0, 44.9, 44.9, 44.9, 44.9, 44.7, 44.5, 44.4, 44.3, 44.3, 44.3, 44.0, 43.9, 43.7, 43.6, 43.6, 43.6, 43.1, 42.5, 42.5, 42.5, 42.5, 42.5, 42.3, 41.8, 41.5, 41.5, 41.5, 41.4, 41.3, 41.2, 41.2, 41.0, 41.0, 41.0, 41.0, 40.9, 40.7, 40.7, 40.5, 40.4, 40.2, 40.1, 40.1, 40.0, 40.0, 39.8, 39.7, 39.6, 39.5, 39.3, 39.0, 39.0, 38.6, 38.5, 38.4, 38.3, 38.0, 38.0, 38.0, 37.5, 37.3, 37.2, 37.1, 37.1, 37.0, 37.0, 37.0, 37.0, 37.0, 36.8, 36.8, 36.7, 36.5, 36.5, 36.5, 36.0, 36.0, 36.0, 36.0, 36.0, 36.0, 35.6, 35.5, 35.4, 35.1, 35.1, 35.0, 34.8, 34.8, 34.8, 34.5, 34.5, 34.3, 33.8, 33.8, 33.7, 33.5, 33.5, 33.2, 33.1, 32.7, 32.4, 32.0, 32.0, 31.6, 31.0, 31.0, 30.9, 30.9, 30.9, 30.5, 30.5, 30.2, 30.0, 30.0, 30.0, 29.8, 29.7, 29.4, 29.3, 29.0, 28.7, 28.5, 28.4, 28.1, 28.0, 28.0, 27.6, 27.5, 27.0, 27.0, 26.9, 26.8, 26.8, 26.7, 26.5, 26.4, 26.0, 26.0, 26.0, 25.7, 25.6, 25.0, 24.9, 24.1, 24.0, 23.6, 23.5, 23.5, 23.5, 23.4, 23.0, 23.0, 23.0, 23.0, 22.7, 22.5, 22.3, 22.0, 22.0, 21.4, 21.3, 21.0, 20.9, 20.9, 20.8, 20.5, 20.4, 20.2, 20.0, 19.5, 19.0, 18.3, 18.0, 18.0, 17.5, 17.4, 17.0, 16.9, 16.6, 16.4, 16.0, 16.0, 16.0, 16.0, 15.2, 15.0, 14.0, 13.7, 13.6, 13.0, 11.0, 11.0, 9.6, 9.0, 8.8, 7.0, 6.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, '.', 'Protein (g)']
After looking through every present data in Protein (g), we see a singular . in our data. This is likely what the 1,690th value is, which is more than likely a typo while working through the data.
Fat (g) will take care of that, along with removing the column names, due to it also being in our columns, making our number of data, 1,688.
First, we'll delete all the uneeded rows in our current dataframe, but we'll have to see what each column corresponds to. Looking through the excel sheet, I started counting every data point, along with dates from cel 478, and will be looking at it from index 470 to 480
In [8]:
df[470:480]
Out[8]:
| Date | Weight (kg) | Waist (cm) | BMI | Unnamed: 10 | Unnamed: 12 | Unnamed: 14 | Fast length (hours) | Fast avg. 7 (hours) | Height | ... | Fat (g) | Carbohydrate % | Protein % | Fat % | Macro score | Score*Log. Kcal | Weight (kg).1 | Waist (cm).1 | Unnamed: 40 | Unnamed: 42 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 470 | March 20th~ | 49.9 | 58.7 | 16.481702 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 471 | NaN | 50.3 | 60.4 | 16.613820 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 472 | NaN | 50.0 | 59.3 | 16.514731 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 473 | NaN | 50.2 | 60.4 | 16.580790 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 474 | NaN | 49.8 | 59.6 | 16.448672 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 475 | NaN | 50.7 | 61.2 | 16.745937 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 476 | NaN | 50.3 | 59.7 | 16.613820 | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 477 | 2020-03-30 00:00:00 | 50.8 | 61.7 | 16.778967 | Kcals accumulated | Daily intake (kcal) | Kcals eaten (week) | Kcal avg. week (day) | Log. Kcal dangers | Carbohydrate (g) | ... | Fat (g) | Carbohydrate % | Protein % | Fat % | Macro score | Score*Log. Kcal | Weight (kg) | Waist (cm) | Days/2 since tracking kcals | Kcal avg. start |
| 478 | 2020-03-30 00:00:00 | 50.2 | 59.7 | 16.580790 | 1424 | 1424 | 1424 | 1424 | 2.037701 | 206.79 | ... | 39.16 | 59.099743 | 15.718777 | 25.18148 | 81.217491 | 143.54043 | 50.2 | 59.7 | 1 | NaN |
| 479 | 2020-03-31 00:00:00 | 50.3 | 61.1 | 16.613820 | 1424 | NaN | 1424 | 1424 | 2.037701 | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | 50.3 | 61.1 | 2 | 1424 |
10 rows × 21 columns
List of columns to remove:¶
BMI- Essentially another column for weight, but in a different unitKcals accumulated+Kcals eaten (week)- Doesn't really provide extra data point for our prediction model. But I decided to keepKcal avg. week (day), showing the average calories consumed for a day, from the previous 7 days.Log. Kcal dangers- Similar to BMI, only ever used to warn myself of overeating, but have stopped using this metric.Macro score- Calculates based on the macro % diet I've set, which is 30% Fat, 30% Protein, 40% Carbs. Lower the number, closer the macros are to target diet.Score*Log Kcal- UsesLog. Kcal dangersmultiplied withMacro score, another redundant feature that I don't use.Weight (kg)+Waist (cm)- Don't need another weight and waist column with the same data.Days/2 since tracking kcals- A form of indexing system for all the new data I've been recording and feature engineering.Kcal avg. start- Measures mean caloric intake, by the number of results I've been recording.
In [9]:
# Remove mentioned rows above from data frame
df = df.drop(columns=['BMI', 'Unnamed: 10', 'Unnamed: 14', 'Fast avg. 7 (hours)', 'Macro score', 'Score*Log. Kcal', 'Weight (kg).1', 'Waist (cm).1', 'Unnamed: 40', 'Unnamed: 42'])
df[470:480]
Out[9]:
| Date | Weight (kg) | Waist (cm) | Unnamed: 12 | Fast length (hours) | Height | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 470 | March 20th~ | 49.9 | 58.7 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 471 | NaN | 50.3 | 60.4 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 472 | NaN | 50.0 | 59.3 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 473 | NaN | 50.2 | 60.4 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 474 | NaN | 49.8 | 59.6 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 475 | NaN | 50.7 | 61.2 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 476 | NaN | 50.3 | 59.7 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 477 | 2020-03-30 00:00:00 | 50.8 | 61.7 | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % |
| 478 | 2020-03-30 00:00:00 | 50.2 | 59.7 | 1424 | 1424 | 206.79 | 55 | 39.16 | 59.099743 | 15.718777 | 25.18148 |
| 479 | 2020-03-31 00:00:00 | 50.3 | 61.1 | NaN | 1424 | NaN | NaN | NaN | NaN | NaN | NaN |
We'll remove index 0 to 476, as it does not have the features we need to predict my weight or waist data.
In [10]:
df = df[477:]
df.head()
Out[10]:
| Date | Weight (kg) | Waist (cm) | Unnamed: 12 | Fast length (hours) | Height | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 477 | 2020-03-30 00:00:00 | 50.8 | 61.7 | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % |
| 478 | 2020-03-30 00:00:00 | 50.2 | 59.7 | 1424 | 1424 | 206.79 | 55 | 39.16 | 59.099743 | 15.718777 | 25.18148 |
| 479 | 2020-03-31 00:00:00 | 50.3 | 61.1 | NaN | 1424 | NaN | NaN | NaN | NaN | NaN | NaN |
| 480 | 2020-03-31 00:00:00 | 50.0 | 59.9 | 1997 | 1710.5 | 283.65 | 38 | 74.8 | 57.893663 | 7.755893 | 34.350444 |
| 481 | 2020-04-01 00:00:00 | 50.8 | 61.0 | NaN | 1710.5 | NaN | NaN | NaN | NaN | NaN | NaN |
Now lets change the title of the column to the one from index 477, then afterwards, removing index 477.
In [11]:
# Rename multiple columns
df = df.rename(columns={
'Unnamed: 12':'Daily intake (kcal)',
'Fast length (hours)':'Kcal avg. week (day)',
'Height':'Carbohydrate (g)'
})
df=df[1:]
df
Out[11]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 478 | 2020-03-30 00:00:00 | 50.2 | 59.7 | 1424 | 1424 | 206.79 | 55 | 39.16 | 59.099743 | 15.718777 | 25.18148 |
| 479 | 2020-03-31 00:00:00 | 50.3 | 61.1 | NaN | 1424 | NaN | NaN | NaN | NaN | NaN | NaN |
| 480 | 2020-03-31 00:00:00 | 50.0 | 59.9 | 1997 | 1710.5 | 283.65 | 38 | 74.8 | 57.893663 | 7.755893 | 34.350444 |
| 481 | 2020-04-01 00:00:00 | 50.8 | 61.0 | NaN | 1710.5 | NaN | NaN | NaN | NaN | NaN | NaN |
| 482 | 2020-04-01 00:00:00 | 50.6 | 60.0 | 901 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 30632 | 2066-07-15 00:00:00 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 30633 | 2066-07-16 00:00:00 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 30634 | 2066-07-16 00:00:00 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 30635 | 2066-07-17 00:00:00 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 30636 | 2066-07-17 00:00:00 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
30159 rows × 11 columns
Looking at Fat (g), you can see how data is entered every 2 cels, and on the date corner, there's a doubling on the sequential dates. So what I want to do, is remove any rows, that contain Nan in Fat (g), as only every 2 column of data is entered. In addition, the weight and waist has a morning and night check-ins, (hence the 2 repeating dates), but have moved towards only morning check-ins to save on time.
In [12]:
# Removing NaN rows, using 'Fat (g)' as our source of valid data.
df = df.dropna(subset=['Fat (g)'])
df
Out[12]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 478 | 2020-03-30 00:00:00 | 50.2 | 59.7 | 1424 | 1424 | 206.79 | 55 | 39.16 | 59.099743 | 15.718777 | 25.18148 |
| 480 | 2020-03-31 00:00:00 | 50.0 | 59.9 | 1997 | 1710.5 | 283.65 | 38 | 74.8 | 57.893663 | 7.755893 | 34.350444 |
| 482 | 2020-04-01 00:00:00 | 50.6 | 60.0 | 901 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 |
| 484 | 2020-04-02 00:00:00 | 50.0 | 58.8 | 1578 | 1475 | 212.28 | 91 | 35.64 | 55.357655 | 23.73067 | 20.911675 |
| 486 | 2020-04-03 00:00:00 | 49.7 | 59.1 | 1870 | 1554 | 232.41 | 120 | 42.24 | 51.940999 | 26.818639 | 21.240362 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 3844 | 2024-11-07 00:00:00 | 73.4 | 74.0 | 2659 | 2911.571429 | 316.3 | 149.1 | 91.5 | 47.119288 | 22.211463 | 30.669249 |
| 3846 | 2024-11-08 00:00:00 | 74.2 | 75.0 | 3509 | 2967 | 263.5 | 198 | 179 | 30.488863 | 22.910038 | 46.601099 |
| 3848 | 2024-11-09 00:00:00 | 74.7 | 74.0 | 3732 | 3105.142857 | 302.3 | 229.5 | 175.5 | 32.622009 | 24.765964 | 42.612027 |
| 3850 | 2024-11-10 00:00:00 | 74.7 | 74.0 | 2812 | 3163.571429 | 247.6 | 162.3 | 121.3 | 36.261121 | 23.768901 | 39.969978 |
| 3852 | 2024-11-11 00:00:00 | 74.7 | 74.0 | 3086 | 3052.571429 | 295.1 | 184 | 118.9 | 39.524527 | 24.644232 | 35.831241 |
1688 rows × 11 columns
We now have 1688 valid data points to work with for this project, but we'll need to reset the index number for the dataframe.
In [13]:
# Reset the index of dataframe
df = df.reset_index(drop=True) # drop=True > drops our old index
df
Out[13]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020-03-30 00:00:00 | 50.2 | 59.7 | 1424 | 1424 | 206.79 | 55 | 39.16 | 59.099743 | 15.718777 | 25.18148 |
| 1 | 2020-03-31 00:00:00 | 50.0 | 59.9 | 1997 | 1710.5 | 283.65 | 38 | 74.8 | 57.893663 | 7.755893 | 34.350444 |
| 2 | 2020-04-01 00:00:00 | 50.6 | 60.0 | 901 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 |
| 3 | 2020-04-02 00:00:00 | 50.0 | 58.8 | 1578 | 1475 | 212.28 | 91 | 35.64 | 55.357655 | 23.73067 | 20.911675 |
| 4 | 2020-04-03 00:00:00 | 49.7 | 59.1 | 1870 | 1554 | 232.41 | 120 | 42.24 | 51.940999 | 26.818639 | 21.240362 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1683 | 2024-11-07 00:00:00 | 73.4 | 74.0 | 2659 | 2911.571429 | 316.3 | 149.1 | 91.5 | 47.119288 | 22.211463 | 30.669249 |
| 1684 | 2024-11-08 00:00:00 | 74.2 | 75.0 | 3509 | 2967 | 263.5 | 198 | 179 | 30.488863 | 22.910038 | 46.601099 |
| 1685 | 2024-11-09 00:00:00 | 74.7 | 74.0 | 3732 | 3105.142857 | 302.3 | 229.5 | 175.5 | 32.622009 | 24.765964 | 42.612027 |
| 1686 | 2024-11-10 00:00:00 | 74.7 | 74.0 | 2812 | 3163.571429 | 247.6 | 162.3 | 121.3 | 36.261121 | 23.768901 | 39.969978 |
| 1687 | 2024-11-11 00:00:00 | 74.7 | 74.0 | 3086 | 3052.571429 | 295.1 | 184 | 118.9 | 39.524527 | 24.644232 | 35.831241 |
1688 rows × 11 columns
In [14]:
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1688 entries, 0 to 1687 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 1688 non-null object 1 Weight (kg) 1688 non-null float64 2 Waist (cm) 1688 non-null float64 3 Daily intake (kcal) 1688 non-null object 4 Kcal avg. week (day) 1688 non-null object 5 Carbohydrate (g) 1688 non-null object 6 Protein (g) 1688 non-null object 7 Fat (g) 1688 non-null object 8 Carbohydrate % 1688 non-null object 9 Protein % 1688 non-null object 10 Fat % 1688 non-null object dtypes: float64(2), object(9) memory usage: 145.2+ KB
Despite all columns except being floats, most of then are objects. Let's convert them into float columns.
In [15]:
# Convert column object into float
convert = ['Daily intake (kcal)', 'Kcal avg. week (day)', 'Carbohydrate (g)', 'Protein (g)', 'Fat (g)', 'Carbohydrate %', 'Protein %', 'Fat %'] # all columns that need to be converted
df[convert] = df[convert].astype(float)
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1688 entries, 0 to 1687 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 1688 non-null object 1 Weight (kg) 1688 non-null float64 2 Waist (cm) 1688 non-null float64 3 Daily intake (kcal) 1688 non-null float64 4 Kcal avg. week (day) 1688 non-null float64 5 Carbohydrate (g) 1688 non-null float64 6 Protein (g) 1688 non-null float64 7 Fat (g) 1688 non-null float64 8 Carbohydrate % 1688 non-null float64 9 Protein % 1688 non-null float64 10 Fat % 1688 non-null float64 dtypes: float64(10), object(1) memory usage: 145.2+ KB
Data interpretation/visualization¶
Now most of the data has been cleaned, we can make some initial interpretations of the data, and visually see them as graphs.
In [16]:
df.describe()
Out[16]:
| Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|
| count | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 |
| mean | 60.912915 | 67.264502 | 2480.717417 | 2477.837438 | 258.473904 | 106.511671 | 111.140604 | 42.339985 | 16.806491 | 40.567946 |
| std | 6.187260 | 4.024980 | 840.246992 | 744.614903 | 112.001458 | 60.502474 | 45.456362 | 13.190117 | 6.866947 | 11.365782 |
| min | 48.600000 | 58.200000 | 0.000000 | 1102.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 |
| 25% | 58.600000 | 64.600000 | 1991.000000 | 2006.357143 | 184.550000 | 61.200000 | 82.100000 | 34.665653 | 11.893795 | 34.012749 |
| 50% | 59.600000 | 66.350000 | 2187.000000 | 2199.071429 | 245.700000 | 91.650000 | 105.600000 | 43.072585 | 16.402331 | 40.153975 |
| 75% | 62.300000 | 70.300000 | 3001.500000 | 2890.357143 | 316.925000 | 146.800000 | 138.100000 | 50.688393 | 20.810325 | 47.398633 |
| max | 76.700000 | 77.900000 | 5231.000000 | 4453.428571 | 696.600000 | 374.400000 | 318.600000 | 100.000000 | 57.626769 | 78.334271 |
Weight vs Waist: The numbers are reasonably similar to each other, which could mean a change in one, is most likely to change the other as well.
Daily intake (kcal): We can see there are data points with no calories consumed. Because I knew the data I put in, it's days where I did extended fasts, and ate nothing within that day.
Carbohydrate %: Max is 100%, which is unusual to only be consuming carbs and no protein and fats. Again, since I knew the data, it was a day when I only drank a bottle of ginger beer. Soft drink with only sugars.
Let's create graphs and visualize the overall change in weight overtime
In [17]:
# Create weight to date graph
fig, ax = plt.subplots(figsize=(20,10)) # set figure size, and create two variables for the figure, and axis within the figure
ax.set(ylabel='Weight (kg)', # set label names for x and y
xlabel='Date')
ax.plot(df['Date'].astype(str), df['Weight (kg)']); # plotting Date (but also converting it into string from object), and Weight
Uh oh, looks like we did the date column wrong, as it was a object, and we simply only converted it as string. Lets change it to datetime value and see if it fixes things.
In [18]:
df['Date'] = pd.to_datetime(df['Date'])
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1688 entries, 0 to 1687 Data columns (total 11 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 1688 non-null datetime64[ns] 1 Weight (kg) 1688 non-null float64 2 Waist (cm) 1688 non-null float64 3 Daily intake (kcal) 1688 non-null float64 4 Kcal avg. week (day) 1688 non-null float64 5 Carbohydrate (g) 1688 non-null float64 6 Protein (g) 1688 non-null float64 7 Fat (g) 1688 non-null float64 8 Carbohydrate % 1688 non-null float64 9 Protein % 1688 non-null float64 10 Fat % 1688 non-null float64 dtypes: datetime64[ns](1), float64(10) memory usage: 145.2 KB
In [19]:
fig, ax = plt.subplots(figsize=(20,10))
ax.set(ylabel='Weight (kg)',
xlabel='Date')
ax.plot(df['Date'], df['Weight (kg)']);
We can see the trends of weight loss, weight gain, and weight maintaining. Let's try and compare it to other data points.
First we'll try waist measurement, and see if our first hypothesis of both being somwhat correlated checks out.
In [20]:
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement overtime', xlabel='Date')
ax.plot(df['Date'], df['Weight (kg)'], label='Weight in kg')
ax.plot(df['Date'], df['Waist (cm)'], label='Waist in cm')
ax.legend();
It is clear that if I gained on either one of the 2 variables, then the the other variable will also increase as well.
I also see how the waist measurement numbers, starts to meet up closely with the weight measurement numbers. I've been consistently going to the gym since Q3 of 2023, and it could mean that building up muscle or lowering body fat has the consequence of minimizing waist circumference. Though I won't truly know the conclusion, as I don't consistently track my body fat %, and neither are the tools I have that accurate to track such data point.
Let's also compare weight with our other features, that aim to predict weight + waist size.
In [301]:
fig, ax1 = plt.subplots(nrows=2, # create 4 different graphs, comparing 4 different columns to the weight data
ncols=2, # ax1 > adding a 1 to it, allows us to create a secondary y axis, due to varying values between 2 different data
figsize=(20,10))
ax1[0,0].set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1[0,0].plot(df['Date'], df['Daily intake (kcal)'], label='Calorie consumption in kcal', color='orange', linewidth=0.5) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1[0,0].twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[0,0].legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend()
# rinse and repeat
ax1[0,1].set(title='Weight and Carbohydrate % consumption overtime', xlabel='Date')
ax1[0,1].plot(df['Date'], df['Carbohydrate %'], label='Carbohydrate consumption %', color='orange', linewidth=0.5)
ax3 = ax1[0,1].twinx()
ax3.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[0,1].legend(loc='lower right')
ax3.legend()
ax1[1,0].set(title='Weight and Protein % consumption overtime', xlabel='Date')
ax1[1,0].plot(df['Date'], df['Protein %'], label='Protein consumption %', color='orange', linewidth=0.5)
ax4 = ax1[1,0].twinx()
ax4.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[1,0].legend(loc='lower center')
ax4.legend()
ax1[1,1].set(title='Weight and Fat % consumption overtime', xlabel='Date')
ax1[1,1].plot(df['Date'], df['Fat %'], label='Fat consumption %', color='orange', linewidth=0.5)
ax5 = ax1[1,1].twinx()
ax5.plot(df['Date'], df['Weight (kg)'], label='Weight in kg', linewidth=0.5)
ax1[1,1].legend(loc='lower center')
ax5.legend();
There's clear correlation between higher intake of calories, leading to more weight gain. And then vice versa with lower caloric intake.
Though macro % consumption is a bit less conclusive than calories. Take carbohydrate %, there's higher percentage in consumption when I was ~50kg and a somwhat decrease within the weight gain in early 2021, which might lead to thinking that higher carbs lead to lower weight. But carbohydrate % consumption stays quite leveled throughout the rest of the chart, and shows varying levels of weight gain and loss, making it quite inconclusive.
Fat % is a bit more inconclusive. We see a rise in fat % consumption, with the rise in weight in early 2021. Then it gradually goes down and flattens itself out for the rest of the graph, seemingly with zero to no correlation to future weight gain or loss.
Maybe protein % has a miniscule change in the weight loss, as the consumption stays somwhat stable, but there's a slight more consumption in protein, where weight has slowly been lost, at the end of the graph.
Graph Anomaly:¶
There is an anomaly in the caloric/weight graph. Near Q3 of 2021, there's an uptake in caloric consumption, yet weight had remained stable or had went down. This was a period of time where I took the chemical, dnp, known for its weight loss effect, and often used by bodybuilders; and must likelly be the cause of the graph anomaly.
Let's look at Q3 and Q4 of 2021, and have a better look at it
In [298]:
# plot Q3 and Q4 2021 (july till december) to better look at anomaly in caloric vs weight graph
anomaly_df = (df['Date'] >= '2021-07-01') & (df['Date'] <= '2021-12-31') # returns True and False, matching to the boolean operation we've given. Is it between july and december?
anomaly = df[anomaly_df] # putting boolean values into dataframe, to than only give us a dataframe, containing True
fig, ax1 = plt.subplots(figsize=(20,5))
ax1.set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1.plot(anomaly['Date'], anomaly['Daily intake (kcal)'], label='Calorie consumption in kcal', color='orange', linewidth=2) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1.twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(anomaly['Date'], anomaly['Weight (kg)'], label='Weight in kg', linewidth=2)
ax1.legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend();
Now we get a better visual of the graph anomaly. Where weight was quite consistent on the days that caloric consumption was ~2000kcal, but notice how weight was loss, despite higher consumption of calories, reaching >3000kcal for the September and October period. Looking back at the original excel sheet, there's a stated dnp dosage column, showing evidence and the amount of dnp that was consumed, between the periods of September and October.
We'll have to remove Sep-2021 and Oct-2021 from our data, to prevent third-party influence from changing our data. This does mean that there'd be a very big jump in weight difference from our previous weight data (08/31) to the next data (11/01), but is better than having anomalies in the data.
In [22]:
# Remove 2021/09 and 2021/10 due to anomaly data that is likely affected from third-party influence
anomaly_date = ~((df['Date'] >= '2021-09-01') & (df['Date'] <= '2021-10-31')) # find 2021/09 and 2021/10, with `not` symbol applied (~)
clean_df = df[anomaly_date].reset_index(drop=True)
clean_df
Out[22]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020-03-30 | 50.2 | 59.7 | 1424.0 | 1424.000000 | 206.79 | 55.0 | 39.16 | 59.099743 | 15.718777 | 25.181480 |
| 1 | 2020-03-31 | 50.0 | 59.9 | 1997.0 | 1710.500000 | 283.65 | 38.0 | 74.80 | 57.893663 | 7.755893 | 34.350444 |
| 2 | 2020-04-01 | 50.6 | 60.0 | 901.0 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 |
| 3 | 2020-04-02 | 50.0 | 58.8 | 1578.0 | 1475.000000 | 212.28 | 91.0 | 35.64 | 55.357655 | 23.730670 | 20.911675 |
| 4 | 2020-04-03 | 49.7 | 59.1 | 1870.0 | 1554.000000 | 232.41 | 120.0 | 42.24 | 51.940999 | 26.818639 | 21.240362 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1622 | 2024-11-07 | 73.4 | 74.0 | 2659.0 | 2911.571429 | 316.30 | 149.1 | 91.50 | 47.119288 | 22.211463 | 30.669249 |
| 1623 | 2024-11-08 | 74.2 | 75.0 | 3509.0 | 2967.000000 | 263.50 | 198.0 | 179.00 | 30.488863 | 22.910038 | 46.601099 |
| 1624 | 2024-11-09 | 74.7 | 74.0 | 3732.0 | 3105.142857 | 302.30 | 229.5 | 175.50 | 32.622009 | 24.765964 | 42.612027 |
| 1625 | 2024-11-10 | 74.7 | 74.0 | 2812.0 | 3163.571429 | 247.60 | 162.3 | 121.30 | 36.261121 | 23.768901 | 39.969978 |
| 1626 | 2024-11-11 | 74.7 | 74.0 | 3086.0 | 3052.571429 | 295.10 | 184.0 | 118.90 | 39.524527 | 24.644232 | 35.831241 |
1627 rows × 11 columns
Feature engineering¶
After seeing how different variables affect weight gain specifically, and that waist size also has heavy correlation to weight as well, there's a big problem with predicting weight and waist. It's not a typical regression problem, where you predict a value of something, but we're predicting the loss and gain of 2 different variables, based on other datapoints.
It's prediction will then be added onto the current weight and waist values, and will continue it's prediction with its previously predicted value. This does make ML accuracy even harder, as we're also relying on our predicted value to then make another prediction.
Anomaly issue¶
Because we got a portion of data removed, there's a substantial difference in weight and possibly waist size between the data points, outside the anomaly time frame. So we'll do our feature engineering on the dataframe, before date anomaly removal, to get the data point difference of its actual neighbour.
In [23]:
# Create features on weight and waist gain/loss
df['Weight Difference'] = df['Weight (kg)'].diff()
df['Waist Difference'] = df['Waist (cm)'].diff()
# Display the DataFrame with the new columns
print(df)
Date Weight (kg) Waist (cm) Daily intake (kcal) \
0 2020-03-30 50.2 59.7 1424.0
1 2020-03-31 50.0 59.9 1997.0
2 2020-04-01 50.6 60.0 901.0
3 2020-04-02 50.0 58.8 1578.0
4 2020-04-03 49.7 59.1 1870.0
... ... ... ... ...
1683 2024-11-07 73.4 74.0 2659.0
1684 2024-11-08 74.2 75.0 3509.0
1685 2024-11-09 74.7 74.0 3732.0
1686 2024-11-10 74.7 74.0 2812.0
1687 2024-11-11 74.7 74.0 3086.0
Kcal avg. week (day) Carbohydrate (g) Protein (g) Fat (g) \
0 1424.000000 206.79 55.0 39.16
1 1710.500000 283.65 38.0 74.80
2 1440.666667 82.35 61.5 29.92
3 1475.000000 212.28 91.0 35.64
4 1554.000000 232.41 120.0 42.24
... ... ... ... ...
1683 2911.571429 316.30 149.1 91.50
1684 2967.000000 263.50 198.0 179.00
1685 3105.142857 302.30 229.5 175.50
1686 3163.571429 247.60 162.3 121.30
1687 3052.571429 295.10 184.0 118.90
Carbohydrate % Protein % Fat % Weight Difference \
0 59.099743 15.718777 25.181480 NaN
1 57.893663 7.755893 34.350444 -0.2
2 38.997017 29.123455 31.879528 0.6
3 55.357655 23.730670 20.911675 -0.6
4 51.940999 26.818639 21.240362 -0.3
... ... ... ... ...
1683 47.119288 22.211463 30.669249 -0.4
1684 30.488863 22.910038 46.601099 0.8
1685 32.622009 24.765964 42.612027 0.5
1686 36.261121 23.768901 39.969978 0.0
1687 39.524527 24.644232 35.831241 0.0
Waist Difference
0 NaN
1 0.2
2 0.1
3 -1.2
4 0.3
... ...
1683 0.1
1684 1.0
1685 -1.0
1686 0.0
1687 0.0
[1688 rows x 13 columns]
In [ ]:
df.info()
<class 'pandas.core.frame.DataFrame'> RangeIndex: 1688 entries, 0 to 1687 Data columns (total 13 columns): # Column Non-Null Count Dtype --- ------ -------------- ----- 0 Date 1688 non-null datetime64[ns] 1 Weight (kg) 1688 non-null float64 2 Waist (cm) 1688 non-null float64 3 Daily intake (kcal) 1688 non-null float64 4 Kcal avg. week (day) 1688 non-null float64 5 Carbohydrate (g) 1688 non-null float64 6 Protein (g) 1688 non-null float64 7 Fat (g) 1688 non-null float64 8 Carbohydrate % 1688 non-null float64 9 Protein % 1688 non-null float64 10 Fat % 1688 non-null float64 11 Weight Difference 1687 non-null float64 12 Waist Difference 1687 non-null float64 dtypes: datetime64[ns](1), float64(12) memory usage: 171.6 KB
In [ ]:
# index 0 has NaN, due to no data point preceding it. We'll change it to = 0
df = df.fillna(0)
In [ ]:
df
Out[ ]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | Weight Difference | Waist Difference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020-03-30 | 50.2 | 59.7 | 1424.0 | 1424.000000 | 206.79 | 55.0 | 39.16 | 59.099743 | 15.718777 | 25.181480 | 0.0 | 0.0 |
| 1 | 2020-03-31 | 50.0 | 59.9 | 1997.0 | 1710.500000 | 283.65 | 38.0 | 74.80 | 57.893663 | 7.755893 | 34.350444 | -0.2 | 0.2 |
| 2 | 2020-04-01 | 50.6 | 60.0 | 901.0 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 | 0.6 | 0.1 |
| 3 | 2020-04-02 | 50.0 | 58.8 | 1578.0 | 1475.000000 | 212.28 | 91.0 | 35.64 | 55.357655 | 23.730670 | 20.911675 | -0.6 | -1.2 |
| 4 | 2020-04-03 | 49.7 | 59.1 | 1870.0 | 1554.000000 | 232.41 | 120.0 | 42.24 | 51.940999 | 26.818639 | 21.240362 | -0.3 | 0.3 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1683 | 2024-11-07 | 73.4 | 74.0 | 2659.0 | 2911.571429 | 316.30 | 149.1 | 91.50 | 47.119288 | 22.211463 | 30.669249 | -0.4 | 0.1 |
| 1684 | 2024-11-08 | 74.2 | 75.0 | 3509.0 | 2967.000000 | 263.50 | 198.0 | 179.00 | 30.488863 | 22.910038 | 46.601099 | 0.8 | 1.0 |
| 1685 | 2024-11-09 | 74.7 | 74.0 | 3732.0 | 3105.142857 | 302.30 | 229.5 | 175.50 | 32.622009 | 24.765964 | 42.612027 | 0.5 | -1.0 |
| 1686 | 2024-11-10 | 74.7 | 74.0 | 2812.0 | 3163.571429 | 247.60 | 162.3 | 121.30 | 36.261121 | 23.768901 | 39.969978 | 0.0 | 0.0 |
| 1687 | 2024-11-11 | 74.7 | 74.0 | 3086.0 | 3052.571429 | 295.10 | 184.0 | 118.90 | 39.524527 | 24.644232 | 35.831241 | 0.0 | 0.0 |
1688 rows × 13 columns
In [297]:
df.describe()
Out[297]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | Weight Difference | Waist Difference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| count | 1688 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1688.000000 | 1687.000000 | 1687.000000 |
| mean | 2022-07-21 11:59:59.999999744 | 60.912915 | 67.264502 | 2480.717417 | 2477.837438 | 258.473904 | 106.511671 | 111.140604 | 42.339985 | 16.806491 | 40.567946 | 0.014523 | 0.008477 |
| min | 2020-03-30 00:00:00 | 48.600000 | 58.200000 | 0.000000 | 1102.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | 0.000000 | -2.500000 | -5.600000 |
| 25% | 2021-05-25 18:00:00 | 58.600000 | 64.600000 | 1991.000000 | 2006.357143 | 184.550000 | 61.200000 | 82.100000 | 34.665653 | 11.893795 | 34.012749 | -0.300000 | -0.400000 |
| 50% | 2022-07-21 12:00:00 | 59.600000 | 66.350000 | 2187.000000 | 2199.071429 | 245.700000 | 91.650000 | 105.600000 | 43.072585 | 16.402331 | 40.153975 | 0.000000 | 0.000000 |
| 75% | 2023-09-16 06:00:00 | 62.300000 | 70.300000 | 3001.500000 | 2890.357143 | 316.925000 | 146.800000 | 138.100000 | 50.688393 | 20.810325 | 47.398633 | 0.300000 | 0.500000 |
| max | 2024-11-11 00:00:00 | 76.700000 | 77.900000 | 5231.000000 | 4453.428571 | 696.600000 | 374.400000 | 318.600000 | 100.000000 | 57.626769 | 78.334271 | 1.500000 | 4.700000 |
| std | NaN | 6.187260 | 4.024980 | 840.246992 | 744.614903 | 112.001458 | 60.502474 | 45.456362 | 13.190117 | 6.866947 | 11.365782 | 0.469796 | 0.824128 |
In [24]:
# Now remove the anomaly dates
anomaly_date = ~((df['Date'] >= '2021-09-01') & (df['Date'] <= '2021-10-31'))
clean_df = df[anomaly_date].reset_index(drop=True)
clean_df
Out[24]:
| Date | Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | Weight Difference | Waist Difference | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 2020-03-30 | 50.2 | 59.7 | 1424.0 | 1424.000000 | 206.79 | 55.0 | 39.16 | 59.099743 | 15.718777 | 25.181480 | NaN | NaN |
| 1 | 2020-03-31 | 50.0 | 59.9 | 1997.0 | 1710.500000 | 283.65 | 38.0 | 74.80 | 57.893663 | 7.755893 | 34.350444 | -0.2 | 0.2 |
| 2 | 2020-04-01 | 50.6 | 60.0 | 901.0 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 | 0.6 | 0.1 |
| 3 | 2020-04-02 | 50.0 | 58.8 | 1578.0 | 1475.000000 | 212.28 | 91.0 | 35.64 | 55.357655 | 23.730670 | 20.911675 | -0.6 | -1.2 |
| 4 | 2020-04-03 | 49.7 | 59.1 | 1870.0 | 1554.000000 | 232.41 | 120.0 | 42.24 | 51.940999 | 26.818639 | 21.240362 | -0.3 | 0.3 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1622 | 2024-11-07 | 73.4 | 74.0 | 2659.0 | 2911.571429 | 316.30 | 149.1 | 91.50 | 47.119288 | 22.211463 | 30.669249 | -0.4 | 0.1 |
| 1623 | 2024-11-08 | 74.2 | 75.0 | 3509.0 | 2967.000000 | 263.50 | 198.0 | 179.00 | 30.488863 | 22.910038 | 46.601099 | 0.8 | 1.0 |
| 1624 | 2024-11-09 | 74.7 | 74.0 | 3732.0 | 3105.142857 | 302.30 | 229.5 | 175.50 | 32.622009 | 24.765964 | 42.612027 | 0.5 | -1.0 |
| 1625 | 2024-11-10 | 74.7 | 74.0 | 2812.0 | 3163.571429 | 247.60 | 162.3 | 121.30 | 36.261121 | 23.768901 | 39.969978 | 0.0 | 0.0 |
| 1626 | 2024-11-11 | 74.7 | 74.0 | 3086.0 | 3052.571429 | 295.10 | 184.0 | 118.90 | 39.524527 | 24.644232 | 35.831241 | 0.0 | 0.0 |
1627 rows × 13 columns
In [25]:
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement overtime', xlabel='Date')
ax.plot(clean_df['Date'], clean_df['Weight Difference'], label='Weight in kg')
ax.plot(clean_df['Date'], clean_df['Waist Difference'], label='Waist in cm')
ax.legend();
Evaluation change¶
Will be changing the prediction of weight and waist size, to the weight and waist difference, as the caloric and macro intake, in addition to the current weight/waist measurement, will have an affect on the change in weight and waist. This project will create to ML models, with one predicting weight difference, and the other for waist difference.
Add datetime parameters for dataframe¶
ML models cannot use datetime values to predict things. So to boost up the number of features that could possibly help with prediction, we will split them up into integers that the ML models can read/use.
In [26]:
# Create separated datetime variables, as integers for ML models to read
ml_df = clean_df.copy() # changing dataframe name to something shorter to rewrite constantly + dulpicating it with .copy()
ml_df['Year'] = ml_df.Date.dt.year
ml_df['Month'] = ml_df.Date.dt.month
ml_df['Day'] = ml_df.Date.dt.day
ml_df['Dayofweek'] = ml_df.Date.dt.dayofweek
ml_df['Dayofyear'] = ml_df.Date.dt.dayofyear
# Drop the date column
ml_df.drop('Date', axis=1, inplace=True) # axis=1 > stating that we're searching by column, not row (0). inplace=True > modifies current dataframe in memory, applying changes while running code
ml_df
Out[26]:
| Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | Weight Difference | Waist Difference | Year | Month | Day | Dayofweek | Dayofyear | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 50.2 | 59.7 | 1424.0 | 1424.000000 | 206.79 | 55.0 | 39.16 | 59.099743 | 15.718777 | 25.181480 | NaN | NaN | 2020 | 3 | 30 | 0 | 90 |
| 1 | 50.0 | 59.9 | 1997.0 | 1710.500000 | 283.65 | 38.0 | 74.80 | 57.893663 | 7.755893 | 34.350444 | -0.2 | 0.2 | 2020 | 3 | 31 | 1 | 91 |
| 2 | 50.6 | 60.0 | 901.0 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 | 0.6 | 0.1 | 2020 | 4 | 1 | 2 | 92 |
| 3 | 50.0 | 58.8 | 1578.0 | 1475.000000 | 212.28 | 91.0 | 35.64 | 55.357655 | 23.730670 | 20.911675 | -0.6 | -1.2 | 2020 | 4 | 2 | 3 | 93 |
| 4 | 49.7 | 59.1 | 1870.0 | 1554.000000 | 232.41 | 120.0 | 42.24 | 51.940999 | 26.818639 | 21.240362 | -0.3 | 0.3 | 2020 | 4 | 3 | 4 | 94 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1622 | 73.4 | 74.0 | 2659.0 | 2911.571429 | 316.30 | 149.1 | 91.50 | 47.119288 | 22.211463 | 30.669249 | -0.4 | 0.1 | 2024 | 11 | 7 | 3 | 312 |
| 1623 | 74.2 | 75.0 | 3509.0 | 2967.000000 | 263.50 | 198.0 | 179.00 | 30.488863 | 22.910038 | 46.601099 | 0.8 | 1.0 | 2024 | 11 | 8 | 4 | 313 |
| 1624 | 74.7 | 74.0 | 3732.0 | 3105.142857 | 302.30 | 229.5 | 175.50 | 32.622009 | 24.765964 | 42.612027 | 0.5 | -1.0 | 2024 | 11 | 9 | 5 | 314 |
| 1625 | 74.7 | 74.0 | 2812.0 | 3163.571429 | 247.60 | 162.3 | 121.30 | 36.261121 | 23.768901 | 39.969978 | 0.0 | 0.0 | 2024 | 11 | 10 | 6 | 315 |
| 1626 | 74.7 | 74.0 | 3086.0 | 3052.571429 | 295.10 | 184.0 | 118.90 | 39.524527 | 24.644232 | 35.831241 | 0.0 | 0.0 | 2024 | 11 | 11 | 0 | 316 |
1627 rows × 17 columns
DATA LEAK ISSUE!!!¶
I've just picked up on an issue between Weight Difference and Weight (kg). The difference is stated from the previous day to the present day. Aka from 50.2kg, to 50.0kg, the weight difference is -0.2kg. This becomes a big problem, as this becomes a loophole for the ML model to get the answer keys. By simply finding the difference of the present and previous days on weight, and then putting that as their prediction, they get the answer. Defying the project's mission on weight and waist prediction!
A similar thing happens with Waist Difference too. There's huge correlation between weight loss and waist loss. Having Waist Difference in the data, makes the ML model too reliant on this specific datapoint to guess Weight Difference, and again, defeats the purpose of the project with weight and waist prediction.
In addition, the end goal is to predict both values of weight and waist. If that's the case, then there is zero possibility that you'd have access to either data on the difference between present and future, as the ML model doesn't know the future!
Solution¶
Fortunately there's a work around, where I could still include Weight (kg) in y_train. By moving the Weight Difference rows by -1 index. What this does, is instead of showing the difference from previous, and present day, it shows the difference between present, and next day! We've got all the data point for the ML model to make a prediction on what my change in weight is going to be on the next day, and avoids the data leakage where the ML model has the ability to cheat the answer from a feature in y_train.
We will also remove the 'Difference' volume, that is not used in X_train, but sits in y_train, as it has a very likely chance of being used by the ML model, to predict the weight difference.
In [27]:
# shift both weight and waist difference by -1 index
ml_df['Weight Difference'] = ml_df['Weight Difference'].shift(-1)
ml_df['Waist Difference'] = ml_df['Waist Difference'].shift(-1)
# fill nan at end of index with 0
ml_df = ml_df.fillna(0)
ml_df
Out[27]:
| Weight (kg) | Waist (cm) | Daily intake (kcal) | Kcal avg. week (day) | Carbohydrate (g) | Protein (g) | Fat (g) | Carbohydrate % | Protein % | Fat % | Weight Difference | Waist Difference | Year | Month | Day | Dayofweek | Dayofyear | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 50.2 | 59.7 | 1424.0 | 1424.000000 | 206.79 | 55.0 | 39.16 | 59.099743 | 15.718777 | 25.181480 | -0.2 | 0.2 | 2020 | 3 | 30 | 0 | 90 |
| 1 | 50.0 | 59.9 | 1997.0 | 1710.500000 | 283.65 | 38.0 | 74.80 | 57.893663 | 7.755893 | 34.350444 | 0.6 | 0.1 | 2020 | 3 | 31 | 1 | 91 |
| 2 | 50.6 | 60.0 | 901.0 | 1440.666667 | 82.35 | 61.5 | 29.92 | 38.997017 | 29.123455 | 31.879528 | -0.6 | -1.2 | 2020 | 4 | 1 | 2 | 92 |
| 3 | 50.0 | 58.8 | 1578.0 | 1475.000000 | 212.28 | 91.0 | 35.64 | 55.357655 | 23.730670 | 20.911675 | -0.3 | 0.3 | 2020 | 4 | 2 | 3 | 93 |
| 4 | 49.7 | 59.1 | 1870.0 | 1554.000000 | 232.41 | 120.0 | 42.24 | 51.940999 | 26.818639 | 21.240362 | 0.4 | 1.2 | 2020 | 4 | 3 | 4 | 94 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 1622 | 73.4 | 74.0 | 2659.0 | 2911.571429 | 316.30 | 149.1 | 91.50 | 47.119288 | 22.211463 | 30.669249 | 0.8 | 1.0 | 2024 | 11 | 7 | 3 | 312 |
| 1623 | 74.2 | 75.0 | 3509.0 | 2967.000000 | 263.50 | 198.0 | 179.00 | 30.488863 | 22.910038 | 46.601099 | 0.5 | -1.0 | 2024 | 11 | 8 | 4 | 313 |
| 1624 | 74.7 | 74.0 | 3732.0 | 3105.142857 | 302.30 | 229.5 | 175.50 | 32.622009 | 24.765964 | 42.612027 | 0.0 | 0.0 | 2024 | 11 | 9 | 5 | 314 |
| 1625 | 74.7 | 74.0 | 2812.0 | 3163.571429 | 247.60 | 162.3 | 121.30 | 36.261121 | 23.768901 | 39.969978 | 0.0 | 0.0 | 2024 | 11 | 10 | 6 | 315 |
| 1626 | 74.7 | 74.0 | 3086.0 | 3052.571429 | 295.10 | 184.0 | 118.90 | 39.524527 | 24.644232 | 35.831241 | 0.0 | 0.0 | 2024 | 11 | 11 | 0 | 316 |
1627 rows × 17 columns
Train test split¶
Let's split the dataset into a train test split. I feel like there's not a whole lot of datapoint I've collected, so this dataset won't include a validation dataset, and the test will be from May 2024 onwards. The point where weight/waist gain start to become a loss.
In [28]:
# Splitting data into test and train datasets
df_test = ml_df[(ml_df.Year == 2024) & (ml_df.Month > 5)]
df_train = ml_df[~((ml_df.Year == 2024) & (ml_df.Month > 5))]
len(df_test), len(df_train)
Out[28]:
(164, 1463)
In [29]:
164/1463
Out[29]:
0.11209842788790157
We will be using about 11% of the data for testing
Now we'll have to split it into x, and y dataset. And into 2 separate x and y's as well for the weight and waist.
We will also remove the corresponding difference column, that is not used by x to prevent unfair advantage to the model.
In addition, I am forecasting my weight, and I realistically wouldn't have a forecast of my waist size on the following day. I only have my current weight, waist, and food consumption from the current day.
In [117]:
# Splitting dataframe into weight and waist, to train 2 ML models
Xweight_train, yweight_train = df_train.drop(['Weight Difference', 'Waist Difference'], axis=1), df_train['Weight Difference']
Xwaist_train, ywaist_train = df_train.drop(['Weight Difference', 'Waist Difference'], axis=1), df_train['Waist Difference']
Xweight_test, yweight_test = df_test.drop(['Weight Difference', 'Waist Difference'], axis=1), df_test['Weight Difference']
Xwaist_test, ywaist_test = df_test.drop(['Weight Difference', 'Waist Difference'], axis=1), df_test['Waist Difference']
# Check the shapes
Xweight_train.shape, yweight_train.shape, Xweight_test.shape, yweight_test.shape
Out[117]:
((1463, 15), (1463,), (164, 15), (164,))
Building evaluation metrics¶
To gauge on accuracy, I'm focusing on the percentage absolute difference between prediction and truth. I don't think some of the metrics work well like MAE (mean absolute error), as the values are usually in the decimals, rather than in whole numbers. Though we will include r^2 and MSE (mean squared error), as to identify the larger errors, since our value variation is so small.
sklearn doesn't have a metric for MAPE (mean absolute percentage error), so we will form one from numpy.
In [55]:
from sklearn.metrics import r2_score
# Create MAPE evaluation metric from numpy
def mean_absolute_percent_error(y_true, y_pred):
y_true = np.ravel(y_true) # Ensures 1D shape
y_pred = np.ravel(y_pred)
y_true, y_pred = np.array(y_true), np.array(y_pred)
dif = np.abs(y_true - y_pred)
percent_error = np.zeros_like(y_true, dtype=float)
# Task1 > If true is 0, and pred is not 0, set to 100%
percent_error[(y_true == 0) & (y_pred != 0)] = 100
# Task2 > If true and pred is 0, set to 0%
percent_error[(y_true == 0) & (y_pred == 0)] = 0
# Task3 > All other cases are then calculated
mask = (y_true != 0) # boolean array of where y_true is 0 or not
percent_error[mask] = (np.abs(y_true[mask] - y_pred[mask]) / np.abs(y_true[mask])) * 100 # absolute dif of true and pred, divided by true. Multiply bvy 100 to get percentage
return np.mean(percent_error)
# also add in mse. r^2 is built in with model scoring
from sklearn.metrics import mean_squared_error
# create function to include all evaluation metric in one space when testing models
def show_scores(model, truth, test, title):
# model predicting values from training
preds = model.predict(test)
truth = np.squeeze(truth)
preds = np.squeeze(preds)
# calculating scores
scores = {
"Test MAPE (%)":mean_absolute_percent_error(truth, preds),
"Test MSE":mean_squared_error(truth, preds),
"Test R^2":r2_score(truth, preds) # model scoring for r^2, done by doing prediction to X, to y
}
return title, scores
Tweaking absolute percent error¶
I thought it'd be easy to form this evaluation metric in a function, but because we're dealing with division, 0's can't be used or I'll cause an error for Infinity. It doesn't help that I'm dealing with very low values, that often hit 0 sometimes. So there's a few tasks to run, in order to avoid all possible forms of errors.
Finding the difference of true, and prediction, then dividing by value of true is the main core of the evaluation. Firstly, we'll have to remove all 0's from True as it'd cause an error. If True and Prediction is 0, then it has no difference, making 0%. Now with True being 0 and Pred being a value, it'll cause infinity. What we'll do is default that to 100%.
It does make it slightly unfair, as for example, True is 0.1, and Prediction is 2, that makes 2000% difference, but is much better than a non-comprehensible value that we can't interpret from.
Algorithm choice¶
We're going to have a look at what is the right algorithm for our regression problem. Python has this cheatsheet to help us discern the right models to use.
- YES - >50 samples
- NO - predicting a category
- YES - predicting a quantity
- YES - <100k samples
- YES - few features should be important
This leaves us with two algorithms, Lasso and ElasticNet, and I will be trying both algorithms to see which performs best overall.
Lasso:¶
Known as Least Absolute Shrinkage and Selection Operator.
Least Absolute means the usage of L1 regularization technique, where we look at the sum of the absolute values (difference) of the coefficients (relationship between the predicting variables, vs the target variable). Shrinkage means shrinking the unimportant features towards zero, in terms of the penalty of L1, in order to reduce complexity of the model, and prevent overfitting.
Selection means selecting only the most significant predictors to the target variable, and excluding less important features in the model, in order to unhance model interpretability and performance. Operator means applying the mathematical operations to L1 penalties to the coefficients during model training.
TLDR: Lasso model both shrinks, by reducing the ability to overfit, and selects, by simplifying the model and excluding irrelevant features¶
ElasticNet:¶
Combination of Lasso Regression and Ridge Regression.
Lasso Regression (L1 Regularization) looks at the penalties, proportional to the difference of value to the coefficients (relationship between prediciting, and targeting variables). Some coefficients reach 0, meaning it has effectively selected this feature. It's chosen due to its likelihood to intersect with the most optimal solution.
Ridge Regression (L2 Regularization) means squaring the coefficients (relationship between the predicting variable, vs the target variable) as its penalty term. It penalizes larger coefficients more heavily due to squaring, but is the opposite with smaller coefficients, in order to prevent model from relying too heavily on any singular feature.
TLDR: Uses L1 and L2 Regularization, striking a balance betwee Ridge and Lasso regression¶
Model Training: Lasso¶
We'll try out the model on Lasso first
In [ ]:
from sklearn.linear_model import Lasso
model_weight = Lasso()
model_weight.fit(Xweight_train, yweight_train)
model_waist = Lasso()
model_waist.fit(Xwaist_train, ywaist_train)
show_scores(model_weight, yweight_test, Xweight_test,'Weight Evaluation'), show_scores(model_waist, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(98.06115173377636),
'Test MSE': 0.35490734435726756,
'Test R^2': 0.03119197823253428}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(101.3501738153358),
'Test MSE': 0.6202001018438599,
'Test R^2': -0.002728014464978612}))
Results¶
MSE and R^2 have very miniscule values. Despite looking really good in terms of evaluatoin value from your typical regression problem, we can't really tell how accurate the predictions really are. However MAPE has provided some good insight over the other two, as we can see its roughly 99.9% different from the truth value. Saying if True is 2, then Prediction could be 4 or 0.
Based on that, the prediction is not really the best, and feels a bit odd that the ML model is predicting at 100% difference from the truth. Could it be defaulting to 0, just because the % difference would likely be lower than whichever prediction they make?
Data scaling¶
One thing suggested with model training, is normalizing/scaling the data. If variable vary way too much from each other, it can cause a bit of confusion with the model training on it. What we do is replace it based on standard deviation, allowing very constant value scaling, and proper balance in contribution.
In [ ]:
# Create standard scaling for our X data
from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()
Xweight_train_scale = scaler.fit_transform(Xweight_train)
Xweight_test_scale = scaler.fit_transform(Xweight_test)
Xwaist_train_scale = scaler.fit_transform(Xwaist_train)
Xwaist_test_scale = scaler.fit_transform(Xwaist_test)
model_weight = Lasso()
model_weight.fit(Xweight_train_scale, yweight_train)
model_waist = Lasso()
model_waist.fit(Xwaist_train_scale, ywaist_train)
show_scores(model_weight, yweight_test, Xweight_test_scale, 'Weight Evaluation'), show_scores(model_waist, ywaist_test, Xwaist_test_scale, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(99.74639365713503),
'Test MSE': 0.3663340273820337,
'Test R^2': -4.741829151555521e-11}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.70550838500678),
'Test MSE': 0.6185494940130513,
'Test R^2': -5.9342357498382015e-05}))
Scaling results¶
Doesn't seem like there's much of an improvement with scaling the data. However the R^2 evaluation is 0, compared to our first model fit. Consulting with chatgpt, it states that the model is perfomring no better than baseline model, predicting the mean of our truth variables. Unable to pick up meaningful relationships between features and target variables.
Gridsearch CV¶
We'll use an adjustment tool, named Gridsearch CV. What it does is, I give it a list of different parameters with values, and the gridsearch does an exhaustive training list on all possible options, to find the best one.
In [ ]:
# Use gridsearch cv to find best parameters for Lasso()
from sklearn.model_selection import GridSearchCV
param_grid = {
'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
'max_iter':[1000,5000,10000,50000], # Iterations for convergence
'tol':[1e-4,1e-3,1e-2,1e-1] # Tolerance for optimization
}
# weight grid search
grid_search_weight = GridSearchCV(
estimator=Lasso(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=2
)
# waist grid search
grid_search_waist = GridSearchCV(
estimator=Lasso(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=2
)
grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)
Fitting 5 folds for each of 96 candidates, totalling 480 fits [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s Fitting 5 folds for each of 96 candidates, totalling 480 fits [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.930e+00, tolerance: 6.430e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.143e+00, tolerance: 8.334e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.930e+00, tolerance: 6.430e-01 model = cd_fast.enet_coordinate_descent(
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.124e-01, tolerance: 6.430e-02 model = cd_fast.enet_coordinate_descent(
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.001e-01, tolerance: 6.430e-02 model = cd_fast.enet_coordinate_descent(
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.2s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s
Out[ ]:
(GridSearchCV(cv=5, estimator=Lasso(),
param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
'max_iter': [1000, 5000, 10000, 50000],
'tol': [0.0001, 0.001, 0.01, 0.1]},
scoring='r2', verbose=2),
GridSearchCV(cv=5, estimator=Lasso(),
param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
'max_iter': [1000, 5000, 10000, 50000],
'tol': [0.0001, 0.001, 0.01, 0.1]},
scoring='r2', verbose=2))
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(98.11454723696536),
'Test MSE': 0.3556541293090464,
'Test R^2': 0.029153442644805794}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(98.73011808148473),
'Test MSE': 0.6193944786817607,
'Test R^2': -0.0014254978879302271}))
Hmmm, it seems to provide little improvement in results...
I think out of curiousity sake, let's see what happens if we allow data leak, and shift the Weight and Waist difference back to it's original position, to see what kind of results come of it.
In [118]:
yweight_train = yweight_train.shift(1)
ywaist_train = ywaist_train.shift(1)
# fill nan at end of index with 0
ywaist_train = ywaist_train.fillna(0)
yweight_train = yweight_train.fillna(0)
ywaist_train, yweight_train
Out[118]:
(0 0.0
1 0.2
2 0.1
3 -1.2
4 0.3
...
1458 0.3
1459 0.1
1460 0.1
1461 -1.0
1462 0.5
Name: Waist Difference, Length: 1463, dtype: float64,
0 0.0
1 -0.2
2 0.6
3 -0.6
4 -0.3
...
1458 0.4
1459 -0.4
1460 0.1
1461 -0.7
1462 0.3
Name: Weight Difference, Length: 1463, dtype: float64)
In [ ]:
# Retry GridSearchCV with leaked dataset
from sklearn.model_selection import GridSearchCV
param_grid = {
'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
'max_iter':[1000,5000,10000,50000], # Iterations for convergence
'tol':[1e-4,1e-3,1e-2,1e-1] # Tolerance for optimization
}
# weight grid search
grid_search_weight = GridSearchCV(
estimator=Lasso(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=2
)
# waist grid search
grid_search_waist = GridSearchCV(
estimator=Lasso(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=2
)
grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)
Fitting 5 folds for each of 96 candidates, totalling 480 fits [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s Fitting 5 folds for each of 96 candidates, totalling 480 fits [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.166e+00, tolerance: 6.422e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.057e+01, tolerance: 8.788e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.240e-01, tolerance: 8.334e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.166e+00, tolerance: 6.422e-01 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.057e+01, tolerance: 8.788e-01 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.166e+00, tolerance: 6.422e+00 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.057e+01, tolerance: 8.788e+00 model = cd_fast.enet_coordinate_descent(
[CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=5000, tol=0.0001; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.433e+00, tolerance: 6.422e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.433e+00, tolerance: 6.422e-01 model = cd_fast.enet_coordinate_descent(
[CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.001, max_iter=5000, tol=0.1; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 6.433e+00, tolerance: 6.422e+00 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.751e+00, tolerance: 6.422e-02 model = cd_fast.enet_coordinate_descent(
[CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.751e+00, tolerance: 6.422e-01 model = cd_fast.enet_coordinate_descent(
[CV] END .............alpha=0.001, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=10000, tol=0.1; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.713e-01, tolerance: 6.422e-02 model = cd_fast.enet_coordinate_descent(
[CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.4s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ............alpha=0.001, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.3s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .............alpha=0.001, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..............alpha=0.001, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.001, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.01, max_iter=5000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .............alpha=0.01, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..............alpha=0.01, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...............alpha=0.01, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=0.01, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=0.1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=0.1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=0.1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=0.1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=0.1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=1000, tol=0.1; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ....................alpha=1, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=1, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=1, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=1, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...................alpha=1, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...................alpha=10, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=10000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=10, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=10, max_iter=50000, tol=0.001; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=10, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ..................alpha=10, max_iter=50000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=1000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=1000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=1000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=1000, tol=0.1; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=5000, tol=0.0001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=5000, tol=0.001; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=5000, tol=0.01; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..................alpha=100, max_iter=5000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=10000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=10000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=10000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=10000, tol=0.1; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ..............alpha=100, max_iter=50000, tol=0.0001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ...............alpha=100, max_iter=50000, tol=0.001; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END ................alpha=100, max_iter=50000, tol=0.01; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s [CV] END .................alpha=100, max_iter=50000, tol=0.1; total time= 0.0s
Out[ ]:
(GridSearchCV(cv=5, estimator=Lasso(),
param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
'max_iter': [1000, 5000, 10000, 50000],
'tol': [0.0001, 0.001, 0.01, 0.1]},
scoring='r2', verbose=2),
GridSearchCV(cv=5, estimator=Lasso(),
param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
'max_iter': [1000, 5000, 10000, 50000],
'tol': [0.0001, 0.001, 0.01, 0.1]},
scoring='r2', verbose=2))
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(98.78643106399778),
'Test MSE': 0.3448826776239478,
'Test R^2': 0.058556803731916185}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(129.8633045684746),
'Test MSE': 0.8208388707545833,
'Test R^2': -0.32711705241647127}))
Well it seems to make the results worse as a matter of fact! I guess we'll shift it back to what it was before
In [119]:
yweight_train = yweight_train.shift(1)
ywaist_train = ywaist_train.shift(1)
# fill nan at end of index with 0
ywaist_train = ywaist_train.fillna(0)
yweight_train = yweight_train.fillna(0)
Model Trainig: ElasticNet¶
Time to try out ElasticNet, and see if it helps with model accuracy
In [ ]:
# replace lasso with elasticnet
from sklearn.linear_model import ElasticNet
model_weight = ElasticNet()
model_weight.fit(Xweight_train_scale, yweight_train)
model_waist = ElasticNet()
model_waist.fit(Xwaist_train_scale, ywaist_train)
show_scores(model_weight, yweight_test, Xweight_test_scale, 'Weight Evaluation'), show_scores(model_waist, ywaist_test, Xwaist_test_scale, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(99.7529524418643),
'Test MSE': 0.36633419215872726,
'Test R^2': -4.498464576396799e-07}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.69630552203824),
'Test MSE': 0.6185537518838363,
'Test R^2': -6.622640396258817e-05}))
Seems to be no different than Lasso when its fit on, right out of the bag. Let's try it with GridSearchCV, and see how well it compares to Lasso's GridSearchCV.
In [ ]:
# Try GridSearchCV with ElasticNet
param_grid = {
'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
'l1_ratio':[0.1,0.3,0.5,0.7,1] # Mix between Lasso and Ridge
}
# weight grid search
grid_search_weight = GridSearchCV(
estimator=ElasticNet(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=2
)
# waist grid search
grid_search_waist = GridSearchCV(
estimator=ElasticNet(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=2
)
grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)
Fitting 5 folds for each of 30 candidates, totalling 150 fits [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s
[CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 2.623e+01, tolerance: 2.302e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 3.713e+01, tolerance: 2.572e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 5.096e+01, tolerance: 2.620e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.972e+01, tolerance: 2.192e-02 model = cd_fast.enet_coordinate_descent(
[CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s Fitting 5 folds for each of 30 candidates, totalling 150 fits [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.1; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.3; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.5; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.730e+02, tolerance: 6.421e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 8.849e+00, tolerance: 7.045e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.347e+02, tolerance: 8.373e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.093e+02, tolerance: 8.786e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 4.978e+00, tolerance: 8.333e-02 model = cd_fast.enet_coordinate_descent( C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\sklearn\linear_model\_coordinate_descent.py:695: ConvergenceWarning: Objective did not converge. You might want to increase the number of iterations, check the scale of the features or consider increasing regularisation. Duality gap: 1.975e+01, tolerance: 6.421e-02 model = cd_fast.enet_coordinate_descent(
[CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ..........................alpha=0.001, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.001, l1_ratio=1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.1; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.3; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.5; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END ...........................alpha=0.01, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=0.01, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=0.1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=0.1, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.1; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.3; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.5; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=1, l1_ratio=0.7; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END ................................alpha=1, l1_ratio=1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.1; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.3; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.5; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END .............................alpha=10, l1_ratio=0.7; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ...............................alpha=10, l1_ratio=1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.1; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.3; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.5; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ............................alpha=100, l1_ratio=0.7; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s [CV] END ..............................alpha=100, l1_ratio=1; total time= 0.0s
Out[ ]:
(GridSearchCV(cv=5, estimator=ElasticNet(),
param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
'l1_ratio': [0.1, 0.3, 0.5, 0.7, 1]},
scoring='r2', verbose=2),
GridSearchCV(cv=5, estimator=ElasticNet(),
param_grid={'alpha': [0.001, 0.01, 0.1, 1, 10, 100],
'l1_ratio': [0.1, 0.3, 0.5, 0.7, 1]},
scoring='r2', verbose=2))
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(98.94022717664029),
'Test MSE': 0.36639817179805406,
'Test R^2': -0.00017509821255967672}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.69630552203824),
'Test MSE': 0.6185537518838363,
'Test R^2': -6.622640396258817e-05}))
Alright, something seems too fishy with the way things are playing out. Let's take a look out what values the model is predicting instead, and see if our hypothesis of the model predicting just 0s being correct.
In [ ]:
# have a peek at the prediction results
yweight_pred = grid_search_weight.best_estimator_.predict(Xweight_test)
ywaist_pred = grid_search_waist.best_estimator_.predict(Xwaist_test)
yweight_pred[:10], ywaist_pred[:10]
Out[ ]:
(array([0.03652816, 0.03547674, 0.03509221, 0.03596346, 0.03993248,
0.04042726, 0.03794259, 0.03827334, 0.03912308, 0.03928173]),
array([0.0112782, 0.0112782, 0.0112782, 0.0112782, 0.0112782, 0.0112782,
0.0112782, 0.0112782, 0.0112782, 0.0112782]))
And why not plot it out for the heck of it as well with the true labels.
In [ ]:
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
Oops, we broke the graph becuase the y test data still has the old index values. Let's remove it.
In [ ]:
# Reset index for y test data
yweight_test = yweight_test.reset_index(drop=True)
ywaist_test = ywaist_test.reset_index(drop=True)
In [ ]:
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
Regardless, it's very clear that the ML model is not doing much in its way to try and predict the value, and it seems to be more so for the waist prediction, where the value just stays consistent all the way!
There must be an issue here with the way my data is set up. Maybe it's just because after all the values are averaged out, it hovers right at 0? Let's check out the mean.
In [120]:
# look at mean of weight and waist difference
print("Mean of Weight Difference:", yweight_train.mean())
print("Mean of Waist Difference:", ywaist_train.mean())
Mean of Weight Difference: 0.015447710184552287 Mean of Waist Difference: 0.011278195488721804
It is very close to 0. ChatGPT suggested, that maybe upscaling the difference can help the model learn, as too many values hovering right at 0, gives little to no opportunities for the model to learn.
So let's try multiplying our weight difference by 100
In [184]:
# multiply our weight and waist difference by 100
yweight_train = yweight_train*100
ywaist_train = ywaist_train*100
yweight_test = yweight_test*100
ywaist_test = ywaist_test*100
yweight_train, ywaist_test
Out[184]:
(0 0.0
1 0.0
2 -20.0
3 60.0
4 -60.0
...
1458 60.0
1459 40.0
1460 -40.0
1461 10.0
1462 -70.0
Name: Weight Difference, Length: 1463, dtype: float64,
1463 80.0
1464 60.0
1465 -20.0
1466 -40.0
1467 10.0
...
1622 100.0
1623 -100.0
1624 0.0
1625 0.0
1626 0.0
Name: Waist Difference, Length: 164, dtype: float64)
Let's try it with the best estimators of ElasticNet
In [ ]:
# Try GridSearchCV with ElasticNet
param_grid = {
'alpha':[0.001,0.01,0.1,1,10,100], # Regularization strength
'l1_ratio':[0.1,0.3,0.5,0.7,1] # Mix between Lasso and Ridge
}
# weight grid search
grid_search_weight = GridSearchCV(
estimator=ElasticNet(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=1
)
# waist grid search
grid_search_waist = GridSearchCV(
estimator=ElasticNet(),
param_grid=param_grid,
scoring='r2',
cv=5,
verbose=1
)
grid_search_weight.fit(Xweight_train, yweight_train), grid_search_waist.fit(Xwaist_train, ywaist_train)
In [ ]:
show_scores(grid_search_weight.best_estimator_, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(grid_search_waist.best_estimator_, ywaist_test, Xwaist_test, 'Waist Evaluation')
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(98.99093421660329),
'Test MSE': 3680.551799975127,
'Test R^2': -0.004698314937412906}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(97.80167748816821),
'Test MSE': 6221.224892910559,
'Test R^2': -0.005836094812318837}))
Meh, didn't do any better. But let's still see it's prediction against the true value
In [ ]:
yweight_pred = grid_search_weight.best_estimator_.predict(Xweight_test)
ywaist_pred = grid_search_waist.best_estimator_.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
At least amplifying the difference by 100 seems to help just a little bit in having varied predictions. So it's time to try something else, and I figured that TensorFlow could be a good start!
Model Training: TensorFlow¶
What's really good with TensorFlow, is it can patterns in non-linear fashion, while ElasticNet learns its value linearly (which is not the case for our data).
Our problem is a regression problem, so ChatGPT suggests we have:
- Have dense (fully connected) layers
- Use an activation function that works well with regression
- End with a single neuron output (for regression)
Activation functions:
Relu-> Best for hidden layers: helps with non-linearityLinear-> Best for output layers: as there's no transformation neededAdam Optimizer-> Good default choice for regression tasks: Adapts learning rates automatically, faster convergence, avoids getting stuck in plateaus, and works well with noisy data.
Let's create our first model, using Relu, Linear, and Adam Optimizer.
In [34]:
pip install tensorflow
Requirement already satisfied: tensorflow in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (2.18.0) Requirement already satisfied: tensorflow-intel==2.18.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow) (2.18.0) Requirement already satisfied: absl-py>=1.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.1.0) Requirement already satisfied: astunparse>=1.6.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.6.3) Requirement already satisfied: flatbuffers>=24.3.25 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (25.2.10) Requirement already satisfied: gast!=0.5.0,!=0.5.1,!=0.5.2,>=0.2.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.6.0) Requirement already satisfied: google-pasta>=0.1.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.2.0) Requirement already satisfied: libclang>=13.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (18.1.1) Requirement already satisfied: opt-einsum>=2.3.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (3.4.0) Requirement already satisfied: packaging in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (24.2) Requirement already satisfied: protobuf!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<6.0.0dev,>=3.20.3 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (5.29.3) Requirement already satisfied: requests<3,>=2.21.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.32.3) Requirement already satisfied: setuptools in c:\program files\windowsapps\pythonsoftwarefoundation.python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\lib\site-packages (from tensorflow-intel==2.18.0->tensorflow) (65.5.0) Requirement already satisfied: six>=1.12.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.17.0) Requirement already satisfied: termcolor>=1.1.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.5.0) Requirement already satisfied: typing-extensions>=3.6.6 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (4.12.2) Requirement already satisfied: wrapt>=1.11.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.17.2) Requirement already satisfied: grpcio<2.0,>=1.24.3 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (1.70.0) Requirement already satisfied: tensorboard<2.19,>=2.18 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.18.0) Requirement already satisfied: keras>=3.5.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (3.8.0) Requirement already satisfied: numpy<2.1.0,>=1.26.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (2.0.2) Requirement already satisfied: h5py>=3.11.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (3.12.1) Requirement already satisfied: ml-dtypes<0.5.0,>=0.4.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.4.1) Requirement already satisfied: tensorflow-io-gcs-filesystem>=0.23.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorflow-intel==2.18.0->tensorflow) (0.31.0) Requirement already satisfied: wheel<1.0,>=0.23.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from astunparse>=1.6.0->tensorflow-intel==2.18.0->tensorflow) (0.45.1) Requirement already satisfied: rich in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (13.9.4) Requirement already satisfied: namex in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (0.0.8) Requirement already satisfied: optree in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (0.14.0) Requirement already satisfied: charset-normalizer<4,>=2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (3.4.1) Requirement already satisfied: idna<4,>=2.5 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (3.10) Requirement already satisfied: urllib3<3,>=1.21.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (2.3.0) Requirement already satisfied: certifi>=2017.4.17 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from requests<3,>=2.21.0->tensorflow-intel==2.18.0->tensorflow) (2025.1.31) Requirement already satisfied: markdown>=2.6.8 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (3.7) Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (0.7.2) Requirement already satisfied: werkzeug>=1.0.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (3.1.3) Requirement already satisfied: MarkupSafe>=2.1.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from werkzeug>=1.0.1->tensorboard<2.19,>=2.18->tensorflow-intel==2.18.0->tensorflow) (3.0.2) Requirement already satisfied: markdown-it-py>=2.2.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from rich->keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (3.0.0) Requirement already satisfied: pygments<3.0.0,>=2.13.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from rich->keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (2.19.1) Requirement already satisfied: mdurl~=0.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from markdown-it-py>=2.2.0->rich->keras>=3.5.0->tensorflow-intel==2.18.0->tensorflow) (0.1.2) Note: you may need to restart the kernel to use updated packages.
[notice] A new release of pip is available: 24.0 -> 25.0.1 [notice] To update, run: C:\Users\deanc\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip
In [35]:
import tensorflow as tf
MODEL 1¶
In [37]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
model_1 = keras.Sequential([
layers.Dense(1, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(1, activation='linear')
])
model_1.compile(optimizer='adam',
loss='mape',
metrics=['mae'])
modelweight_1 = model_1
modelwaist_1 = model_1
modelweight_1.fit(Xweight_train, yweight_train, epochs=100), modelwaist_1.fit(Xwaist_train, ywaist_train, epochs=100)
Epoch 1/100
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
46/46 ━━━━━━━━━━━━━━━━━━━━ 1s 2ms/step - loss: 814765760.0000 - mae: 4.6365 Epoch 2/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50534888.0000 - mae: 0.5537 Epoch 3/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 28712.1465 - mae: 0.4294 Epoch 4/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 34377.3086 - mae: 0.4289 Epoch 5/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18265.8555 - mae: 0.4289 Epoch 6/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 47448.5586 - mae: 0.4289 Epoch 7/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 69117.7266 - mae: 0.4290 Epoch 8/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 21261.1113 - mae: 0.4289 Epoch 9/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 30248.6875 - mae: 0.4289 Epoch 10/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47582.1367 - mae: 0.4289 Epoch 11/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 69111.7656 - mae: 0.4290 Epoch 12/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 21224.6484 - mae: 0.4289 Epoch 13/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26699.9727 - mae: 0.4289 Epoch 14/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 27917.8438 - mae: 0.4289 Epoch 15/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31166.2012 - mae: 0.4289 Epoch 16/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 14923.9990 - mae: 0.4289 Epoch 17/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18820.5586 - mae: 0.4289 Epoch 18/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 27072.0430 - mae: 0.4289 Epoch 19/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26369.6387 - mae: 0.4289 Epoch 20/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50906.2383 - mae: 0.4290 Epoch 21/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47244.7109 - mae: 0.4289 Epoch 22/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26295.0137 - mae: 0.4289 Epoch 23/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 34885.7734 - mae: 0.4289 Epoch 24/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 36128.2148 - mae: 0.4289 Epoch 25/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50980.9727 - mae: 0.4290 Epoch 26/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 14363.5986 - mae: 0.4289 Epoch 27/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 35372.1680 - mae: 0.4289 Epoch 28/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29165.3652 - mae: 0.4289 Epoch 29/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26405.4590 - mae: 0.4289 Epoch 30/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50964.0859 - mae: 0.4290 Epoch 31/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47349.0234 - mae: 0.4289 Epoch 32/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 26348.5039 - mae: 0.4289 Epoch 33/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 34790.0625 - mae: 0.4289 Epoch 34/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 36128.8906 - mae: 0.4289 Epoch 35/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 50901.3320 - mae: 0.4290 Epoch 36/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 14401.1436 - mae: 0.4289 Epoch 37/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 19837.3184 - mae: 0.4289 Epoch 38/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47385.5586 - mae: 0.4290 Epoch 39/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 69208.2109 - mae: 0.4290 Epoch 40/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 22232.1621 - mae: 0.4289 Epoch 41/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32560.6211 - mae: 0.4289 Epoch 42/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 9653.5391 - mae: 0.4289 Epoch 43/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 28403.5703 - mae: 0.4289 Epoch 44/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 33961.5703 - mae: 0.4289 Epoch 45/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29111.8223 - mae: 0.4289 Epoch 46/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24103.3867 - mae: 0.4289 Epoch 47/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 12425.9824 - mae: 0.4289 Epoch 48/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 33108.3711 - mae: 0.4289 Epoch 49/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92331.6953 - mae: 0.4290 Epoch 50/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24899.2012 - mae: 0.4289 Epoch 51/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29133.5195 - mae: 0.4289 Epoch 52/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94018.1641 - mae: 0.4290 Epoch 53/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31684.0586 - mae: 0.4289 Epoch 54/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15133.7920 - mae: 0.4289 Epoch 55/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32233.0938 - mae: 0.4289 Epoch 56/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29290.6699 - mae: 0.4289 Epoch 57/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92252.6641 - mae: 0.4290 Epoch 58/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.9785 - mae: 0.4289 Epoch 59/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29132.3145 - mae: 0.4289 Epoch 60/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94016.2969 - mae: 0.4290 Epoch 61/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31682.7598 - mae: 0.4289 Epoch 62/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15133.8213 - mae: 0.4289 Epoch 63/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32231.5957 - mae: 0.4289 Epoch 64/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29289.7793 - mae: 0.4289 Epoch 65/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92250.4297 - mae: 0.4290 Epoch 66/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.7910 - mae: 0.4289 Epoch 67/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29131.5098 - mae: 0.4289 Epoch 68/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94015.3047 - mae: 0.4290 Epoch 69/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31682.0977 - mae: 0.4289 Epoch 70/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15133.9258 - mae: 0.4289 Epoch 71/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32230.4707 - mae: 0.4289 Epoch 72/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29289.2168 - mae: 0.4289 Epoch 73/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 92248.6719 - mae: 0.4290 Epoch 74/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.7344 - mae: 0.4289 Epoch 75/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.9590 - mae: 0.4289 Epoch 76/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94014.9609 - mae: 0.4290 Epoch 77/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31681.8516 - mae: 0.4289 Epoch 78/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15134.0859 - mae: 0.4289 Epoch 79/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32229.6016 - mae: 0.4289 Epoch 80/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29288.8418 - mae: 0.4289 Epoch 81/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92247.2109 - mae: 0.4290 Epoch 82/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.7559 - mae: 0.4289 Epoch 83/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.5742 - mae: 0.4289 Epoch 84/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94014.9766 - mae: 0.4290 Epoch 85/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31681.8750 - mae: 0.4289 Epoch 86/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15134.2803 - mae: 0.4289 Epoch 87/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32228.8945 - mae: 0.4289 Epoch 88/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29288.6172 - mae: 0.4289 Epoch 89/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92245.9766 - mae: 0.4290 Epoch 90/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.8242 - mae: 0.4289 Epoch 91/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.3047 - mae: 0.4289 Epoch 92/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94015.2344 - mae: 0.4290 Epoch 93/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 31682.0371 - mae: 0.4289 Epoch 94/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 15134.4717 - mae: 0.4289 Epoch 95/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 32228.3145 - mae: 0.4289 Epoch 96/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29288.4707 - mae: 0.4289 Epoch 97/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 92244.8906 - mae: 0.4290 Epoch 98/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 24898.9199 - mae: 0.4289 Epoch 99/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29130.0918 - mae: 0.4289 Epoch 100/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 94015.6250 - mae: 0.4290 Epoch 1/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 51684.5547 - mae: 0.6845 Epoch 2/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 37540.6523 - mae: 0.6845 Epoch 3/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 10654.8145 - mae: 0.6845 Epoch 4/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 29097.0879 - mae: 0.6845 Epoch 5/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18858.0938 - mae: 0.6845 Epoch 6/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18007.2969 - mae: 0.6845 Epoch 7/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 19149.3672 - mae: 0.6845 Epoch 8/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 35210.2695 - mae: 0.6845 Epoch 9/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 38894.6602 - mae: 0.6845 Epoch 10/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 28572.7109 - mae: 0.6845 Epoch 11/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18018.6562 - mae: 0.6845 Epoch 12/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 18118.5293 - mae: 0.6845 Epoch 13/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 27053.2246 - mae: 0.6845 Epoch 14/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47434.3789 - mae: 0.6845 Epoch 15/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 22458.3887 - mae: 0.6845 Epoch 16/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 17581.8223 - mae: 0.6845 Epoch 17/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 47249.5391 - mae: 0.6845 Epoch 18/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60501.5117 - mae: 0.6845 Epoch 19/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60025.2148 - mae: 0.6845 Epoch 20/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61765.7344 - mae: 0.6845 Epoch 21/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60195.4336 - mae: 0.6845 Epoch 22/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61936.9453 - mae: 0.6845 Epoch 23/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 60359.7031 - mae: 0.6845 Epoch 24/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62094.1875 - mae: 0.6845 Epoch 25/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60510.5898 - mae: 0.6845 Epoch 26/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62238.5469 - mae: 0.6845 Epoch 27/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60649.1172 - mae: 0.6845 Epoch 28/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62371.0273 - mae: 0.6845 Epoch 29/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60776.2383 - mae: 0.6845 Epoch 30/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62492.5547 - mae: 0.6845 Epoch 31/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60892.8125 - mae: 0.6845 Epoch 32/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62603.9414 - mae: 0.6845 Epoch 33/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 60999.6836 - mae: 0.6845 Epoch 34/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62706.0312 - mae: 0.6845 Epoch 35/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61097.5977 - mae: 0.6845 Epoch 36/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62799.5312 - mae: 0.6845 Epoch 37/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61187.2891 - mae: 0.6845 Epoch 38/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62885.1328 - mae: 0.6845 Epoch 39/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61269.4141 - mae: 0.6844 Epoch 40/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62963.4805 - mae: 0.6845 Epoch 41/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61344.5898 - mae: 0.6844 Epoch 42/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63035.1680 - mae: 0.6845 Epoch 43/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61413.3750 - mae: 0.6844 Epoch 44/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63100.7383 - mae: 0.6845 Epoch 45/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61476.3242 - mae: 0.6844 Epoch 46/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63160.6953 - mae: 0.6844 Epoch 47/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61533.8945 - mae: 0.6844 Epoch 48/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63215.5195 - mae: 0.6844 Epoch 49/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61586.5273 - mae: 0.6844 Epoch 50/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63265.6172 - mae: 0.6844 Epoch 51/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61634.6523 - mae: 0.6844 Epoch 52/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63311.4102 - mae: 0.6844 Epoch 53/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61678.6328 - mae: 0.6844 Epoch 54/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63353.2422 - mae: 0.6844 Epoch 55/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61718.8359 - mae: 0.6844 Epoch 56/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63391.4492 - mae: 0.6844 Epoch 57/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61755.5664 - mae: 0.6844 Epoch 58/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63426.3398 - mae: 0.6844 Epoch 59/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 61789.1445 - mae: 0.6843 Epoch 60/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63458.2070 - mae: 0.6844 Epoch 61/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61819.8086 - mae: 0.6843 Epoch 62/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63487.2969 - mae: 0.6843 Epoch 63/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61847.8516 - mae: 0.6843 Epoch 64/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63513.8555 - mae: 0.6843 Epoch 65/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61873.4414 - mae: 0.6843 Epoch 66/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63538.0781 - mae: 0.6843 Epoch 67/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61896.8164 - mae: 0.6843 Epoch 68/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 63560.2070 - mae: 0.6843 Epoch 69/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61918.1719 - mae: 0.6843 Epoch 70/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63580.3945 - mae: 0.6843 Epoch 71/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 61937.6836 - mae: 0.6842 Epoch 72/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63598.7969 - mae: 0.6842 Epoch 73/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61955.5039 - mae: 0.6842 Epoch 74/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63615.5977 - mae: 0.6842 Epoch 75/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61971.7734 - mae: 0.6842 Epoch 76/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63630.9180 - mae: 0.6842 Epoch 77/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 61986.6133 - mae: 0.6842 Epoch 78/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 63644.8828 - mae: 0.6842 Epoch 79/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62000.1719 - mae: 0.6841 Epoch 80/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63657.6211 - mae: 0.6841 Epoch 81/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62012.5703 - mae: 0.6841 Epoch 82/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63669.2305 - mae: 0.6841 Epoch 83/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62023.8750 - mae: 0.6841 Epoch 84/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63679.8203 - mae: 0.6841 Epoch 85/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62034.2148 - mae: 0.6840 Epoch 86/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63689.4570 - mae: 0.6840 Epoch 87/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62043.6484 - mae: 0.6840 Epoch 88/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63698.2656 - mae: 0.6840 Epoch 89/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62052.2656 - mae: 0.6840 Epoch 90/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63706.2891 - mae: 0.6840 Epoch 91/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62060.1328 - mae: 0.6839 Epoch 92/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63713.5898 - mae: 0.6839 Epoch 93/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62067.3242 - mae: 0.6839 Epoch 94/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63720.2305 - mae: 0.6839 Epoch 95/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62073.8789 - mae: 0.6838 Epoch 96/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63726.2891 - mae: 0.6838 Epoch 97/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 62079.8828 - mae: 0.6838 Epoch 98/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 1ms/step - loss: 63731.7891 - mae: 0.6838 Epoch 99/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 62085.3672 - mae: 0.6837 Epoch 100/100 46/46 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step - loss: 63736.8125 - mae: 0.6837
Out[37]:
(<keras.src.callbacks.history.History at 0x1aecf17a710>, <keras.src.callbacks.history.History at 0x1aecf172f90>)
In [41]:
show_scores(modelweight_1, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_1, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[41]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(475.9956588090121),
'Test MSE': 168.39310068358228,
'Test R^2': -458.6709235420203}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(195.89982110985835),
'Test MSE': 168.02178535139865,
'Test R^2': -270.6545042662456}))
Tensorflow 2D array¶
One thing different with Lasso and ElasticNet, is it returns predictions in a 1D array such as ((n,)) compared to TensorFlow that make predictions in 2D array such as (n,1).
To fix this, we will need to revise the show_scores function, and use the np.squeeze() function inside it, to set the array as a 1D.
In [39]:
# Create MAPE evaluation metric from numpy
def mean_absolute_percent_error(y_true, y_pred):
y_true, y_pred = np.array(y_true), np.array(y_pred)
dif = np.abs(y_true - y_pred)
percent_error = np.zeros_like(y_true, dtype=float)
# flatten tensor array
y_pred = np.squeeze(y_pred)
# Task1 > If true is 0, and pred is not 0, set to 100%
percent_error[(y_true == 0) & (y_pred != 0)] = 100
# Task2 > If true and pred is 0, set to 0%
percent_error[(y_true == 0) & (y_pred == 0)] = 0
# Task3 > All other cases are then calculated
mask = (y_true != 0) # boolean array of where y_true is 0 or not
percent_error[mask] = (np.abs(y_true[mask] - y_pred[mask]) / np.abs(y_true[mask])) * 100 # absolute dif of true and pred, divided by true. Multiply bvy 100 to get percentage
return np.mean(percent_error)
from sklearn.metrics import r2_score
# also add in mse. r^2 is built in with model scoring
from sklearn.metrics import mean_squared_error
# create function to include all evaluation metric in one space when testing models
def show_scores(model, truth, test, title):
# model predicting values from training
preds = model.predict(test)
# calculating scores
scores = {
"Test MAPE (%)":mean_absolute_percent_error(truth, preds),
"Test MSE":mean_squared_error(truth, preds),
"Test R^2":r2_score(truth,preds) # model scoring for r^2, done by doing prediction to X, to y
}
return title, scores
In [42]:
show_scores(modelweight_1, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_1, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[42]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(475.9956588090121),
'Test MSE': 168.39310068358228,
'Test R^2': -458.6709235420203}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(195.89982110985835),
'Test MSE': 168.02178535139865,
'Test R^2': -270.6545042662456}))
MODEL 2¶
Not good. Let's up our mode layers with more neurons. Increasing by the power of 2 (1, 2, 4, 8, 16, 32)
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
model_2 = keras.Sequential([
layers.Dense(1, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(2, activation='relu'),
layers.Dense(4, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
model_2.compile(optimizer='adam',
loss='mape',
metrics=['mae'])
modelweight_2 = model_2
modelwaist_2 = model_2
modelweight_2.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_2.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd3590b6d0>, <keras.src.callbacks.history.History at 0x2bd35879090>)
In [ ]:
show_scores(modelweight_2, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_2, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(99.99998191581312),
'Test MSE': 3665.918995282517,
'Test R^2': -0.0007039263194963397}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.99996955052484),
'Test MSE': 6185.43143862701,
'Test R^2': -4.9075552263522226e-05}))
MODEL 3¶
Again there's no improvement. How about we reverse the exponent on relu? going 8,4,2,1 instead. We'll also remove the 1 for the relu layer, as it's not helpful to have one singular neuron layer, when there is already one existing specifically for the last layer linear.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
model_3 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(8, activation='relu'),
layers.Dense(4, activation='relu'),
layers.Dense(2, activation='relu'),
layers.Dense(1, activation='linear')
])
model_3.compile(optimizer='adam',
loss='mape',
metrics=['mae'])
modelweight_3 = model_3
modelwaist_3 = model_3
modelweight_3.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_3.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd37836190>, <keras.src.callbacks.history.History at 0x2bd37c25050>)
In [ ]:
show_scores(modelweight_3, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_3, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(99.99996217106948),
'Test MSE': 3665.8529085824352,
'Test R^2': -0.000685886308155359}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.99993630506658),
'Test MSE': 6185.365622942159,
'Test R^2': -3.843459448304998e-05}))
MODEL 4¶
Let's see if changing the loss helps. Maybe I'm too over-reliant on my fabricated function, that it's not working for the data specifically. Let's switch it to MSE
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
model_4 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(8, activation='relu'),
layers.Dense(4, activation='relu'),
layers.Dense(2, activation='relu'),
layers.Dense(1, activation='linear')
])
model_4.compile(optimizer='adam',
loss='mse',
metrics=['mae'])
modelweight_4 = model_4
modelwaist_4 = model_4
modelweight_4.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_4.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd37cb7f50>, <keras.src.callbacks.history.History at 0x2bd379b1f50>)
In [ ]:
show_scores(modelweight_4, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_4, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(99.76027602236368),
'Test MSE': 3672.847703900664,
'Test R^2': -0.0025952899659449358}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.59636176379766),
'Test MSE': 6195.650336744256,
'Test R^2': -0.0017012480347673442}))
MODEL 5¶
Now it looks like it's slightly helpful, but still barely any improvement.
Let's increase neurons in relu, starting from 64
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
model_5 = keras.Sequential([
layers.Dense(64, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
model_5.compile(optimizer='adam',
loss='mse',
metrics=['mae'])
modelweight_5 = model_5
modelwaist_5 = model_5
modelweight_5.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_5.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd37d314d0>, <keras.src.callbacks.history.History at 0x2bd395a0a50>)
In [ ]:
show_scores(modelweight_5, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_5, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(115.64029176907775),
'Test MSE': 4204.008713731997,
'Test R^2': -0.1475889215028252}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(105.77201532508138),
'Test MSE': 6786.071699910744,
'Test R^2': -0.09715947827780091}))
Nevermind, this code performed worse for some reason.
However, I figure it'd be a good time to compare the graph and see what is going on
In [ ]:
yweight_pred = modelweight_5.predict(Xweight_test)
ywaist_pred = modelwaist_5.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
In [ ]:
yweight_pred = modelweight_4.predict(Xweight_test)
ywaist_pred = modelwaist_4.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Okay, interesting results! Despite seeing poorer results for increasing the number of neurons in the model, it has the effect of the ML model trying to effectively learn the pattern of the data, over a simple straight line over 0. Sure we can more easily achieve a low score through getting the average which is close to 0, but we need that risk taking to catch the full dataset!
MODEL 6¶
I remember from TensorFlow 01 extra cirriculars, I did something where I increased exponentially in neurons, then decreased exponentially. Let's give it a go, and increase the top to 256 neurons. (16, 32, 64, 128, 256, 128, 64, 32, 16)
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
model_6 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
model_6.compile(optimizer='adam',
loss='mse',
metrics=['mae'])
modelweight_6 = model_6
modelwaist_6 = model_6
modelweight_6.fit(Xweight_train, yweight_train, epochs=100, verbose=0), modelwaist_6.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
(<keras.src.callbacks.history.History at 0x2bd3ad5ccd0>, <keras.src.callbacks.history.History at 0x2bd3c075050>)
In [ ]:
show_scores(modelweight_6, yweight_test, Xweight_test, 'Weight Evaluation'), show_scores(modelwaist_6, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 29ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Out[ ]:
(('Weight Evaluation',
{'Test MAPE (%)': np.float64(111.76415878255344),
'Test MSE': 4159.561606060438,
'Test R^2': -0.13545597606193827}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(93.5839074131618),
'Test MSE': 6428.471201429294,
'Test R^2': -0.03934329334846254}))
In [ ]:
yweight_pred = modelweight_6.predict(Xweight_test)
ywaist_pred = modelwaist_6.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Increasing the number of neurons for ReLU, does seem to improve the variability in the prediction, which is our goal for now. We need the model to take risks in making some extreme predictions.
Weight and Waist same prediciton¶
One other thing, is I tried to train the model with the same layers, and figured that naming them differently will get them to train differently between the two. This doesn't seem to happen in our predictions, as they're both the same. Let's see if we can fix this by separating them into different codes.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelweight_6 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelweight_6.compile(optimizer='adam',
loss='mse',
metrics=['mae'])
modelweight_6.fit(Xweight_train, yweight_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd3d511610>
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_6 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_6.compile(optimizer='adam',
loss='mse',
metrics=['mae'])
modelwaist_6.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd526a1a50>
In [ ]:
yweight_pred = modelweight_6.predict(Xweight_test)
ywaist_pred = modelwaist_6.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(yweight_pred, label='Weight prediction')
ax.plot(yweight_test, label='True weight')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 20ms/step
Sweet, we got them to both to separate, by training them on separate blocks of code. For now, we'll train on the waist difference data, as both weight and waist have very similar variables and patterns. So theoretically, the model that works well in predicting my waist difference, should work on weight difference as well if I retrain the model on the weight data.
MODEL 7¶
Next step, let's try find the optimal Adam optimizer value. To do so, we'll make a Learning Rate Scheduler, where we exponentially increase the learning rate of Adam. We do this, as learning rate doesn't increase linearly like 0.01, 0.02, 0.03, but exponentially as 0.001, 0.01, 0.1.
In [ ]:
# create optimal learning rate adam optimizer model
tf.random.set_seed(35)
adam_model = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
adam_model.compile(optimizer='adam',
loss='mse',
metrics=['mae'])
# learning rate scheduler, going through different learning rates in exponential increase, based on epoch number
lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10 ** (epoch/20)) # formula for exponential increase
find_lr_history = adam_model.fit(Xwaist_train,
ywaist_train,
epochs=100,
verbose=2,
validation_data=(Xwaist_test,ywaist_test),
callbacks=[lr_scheduler])
Epoch 1/100
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
46/46 - 3s - 58ms/step - loss: 6685.2480 - mae: 58.8719 - val_loss: 6198.9521 - val_mae: 56.1354 - learning_rate: 1.0000e-05 Epoch 2/100 46/46 - 0s - 4ms/step - loss: 6672.3242 - mae: 58.8637 - val_loss: 6194.2646 - val_mae: 56.0520 - learning_rate: 1.1220e-05 Epoch 3/100 46/46 - 0s - 4ms/step - loss: 6671.0337 - mae: 58.8408 - val_loss: 6198.7007 - val_mae: 56.1022 - learning_rate: 1.2589e-05 Epoch 4/100 46/46 - 0s - 4ms/step - loss: 6671.8237 - mae: 58.8759 - val_loss: 6206.0981 - val_mae: 56.1709 - learning_rate: 1.4125e-05 Epoch 5/100 46/46 - 0s - 4ms/step - loss: 6672.6294 - mae: 58.9178 - val_loss: 6212.1694 - val_mae: 56.2194 - learning_rate: 1.5849e-05 Epoch 6/100 46/46 - 0s - 4ms/step - loss: 6673.7681 - mae: 58.9777 - val_loss: 6232.3481 - val_mae: 56.3535 - learning_rate: 1.7783e-05 Epoch 7/100 46/46 - 0s - 4ms/step - loss: 6676.5757 - mae: 59.0470 - val_loss: 6236.3271 - val_mae: 56.4014 - learning_rate: 1.9953e-05 Epoch 8/100 46/46 - 0s - 4ms/step - loss: 6673.7373 - mae: 59.0563 - val_loss: 6255.1958 - val_mae: 56.5190 - learning_rate: 2.2387e-05 Epoch 9/100 46/46 - 0s - 4ms/step - loss: 6675.2412 - mae: 59.1204 - val_loss: 6283.3394 - val_mae: 56.7068 - learning_rate: 2.5119e-05 Epoch 10/100 46/46 - 0s - 4ms/step - loss: 6679.2373 - mae: 59.2138 - val_loss: 6295.4517 - val_mae: 56.8270 - learning_rate: 2.8184e-05 Epoch 11/100 46/46 - 0s - 4ms/step - loss: 6679.3535 - mae: 59.2373 - val_loss: 6314.8853 - val_mae: 57.0322 - learning_rate: 3.1623e-05 Epoch 12/100 46/46 - 0s - 4ms/step - loss: 6680.6235 - mae: 59.2825 - val_loss: 6329.9893 - val_mae: 57.2232 - learning_rate: 3.5481e-05 Epoch 13/100 46/46 - 0s - 4ms/step - loss: 6677.6577 - mae: 59.2895 - val_loss: 6348.1636 - val_mae: 57.4229 - learning_rate: 3.9811e-05 Epoch 14/100 46/46 - 0s - 4ms/step - loss: 6674.8618 - mae: 59.3052 - val_loss: 6360.3604 - val_mae: 57.5080 - learning_rate: 4.4668e-05 Epoch 15/100 46/46 - 0s - 3ms/step - loss: 6669.9160 - mae: 59.2418 - val_loss: 6337.0190 - val_mae: 57.3272 - learning_rate: 5.0119e-05 Epoch 16/100 46/46 - 0s - 7ms/step - loss: 6677.8647 - mae: 59.3522 - val_loss: 6286.3984 - val_mae: 56.8164 - learning_rate: 5.6234e-05 Epoch 17/100 46/46 - 0s - 4ms/step - loss: 6688.3652 - mae: 59.3248 - val_loss: 6215.1484 - val_mae: 56.2655 - learning_rate: 6.3096e-05 Epoch 18/100 46/46 - 0s - 4ms/step - loss: 6688.8496 - mae: 59.1789 - val_loss: 6219.3101 - val_mae: 56.2795 - learning_rate: 7.0795e-05 Epoch 19/100 46/46 - 0s - 4ms/step - loss: 6683.4482 - mae: 59.1189 - val_loss: 6200.3442 - val_mae: 56.0701 - learning_rate: 7.9433e-05 Epoch 20/100 46/46 - 0s - 4ms/step - loss: 6684.1421 - mae: 59.2445 - val_loss: 6221.9268 - val_mae: 56.3175 - learning_rate: 8.9125e-05 Epoch 21/100 46/46 - 0s - 4ms/step - loss: 6672.0439 - mae: 58.9549 - val_loss: 6194.8135 - val_mae: 55.9934 - learning_rate: 1.0000e-04 Epoch 22/100 46/46 - 0s - 4ms/step - loss: 6662.1108 - mae: 58.8144 - val_loss: 6192.9604 - val_mae: 55.9525 - learning_rate: 1.1220e-04 Epoch 23/100 46/46 - 0s - 4ms/step - loss: 6660.2031 - mae: 58.7288 - val_loss: 6193.5547 - val_mae: 55.9644 - learning_rate: 1.2589e-04 Epoch 24/100 46/46 - 0s - 4ms/step - loss: 6660.9663 - mae: 58.7289 - val_loss: 6192.8857 - val_mae: 55.9468 - learning_rate: 1.4125e-04 Epoch 25/100 46/46 - 0s - 4ms/step - loss: 6661.2769 - mae: 58.7287 - val_loss: 6196.5107 - val_mae: 56.0061 - learning_rate: 1.5849e-04 Epoch 26/100 46/46 - 0s - 3ms/step - loss: 6663.8188 - mae: 58.7561 - val_loss: 6195.3076 - val_mae: 55.9926 - learning_rate: 1.7783e-04 Epoch 27/100 46/46 - 0s - 3ms/step - loss: 6664.7349 - mae: 58.7807 - val_loss: 6198.1777 - val_mae: 56.0418 - learning_rate: 1.9953e-04 Epoch 28/100 46/46 - 0s - 4ms/step - loss: 6665.1123 - mae: 58.8003 - val_loss: 6199.1777 - val_mae: 56.0512 - learning_rate: 2.2387e-04 Epoch 29/100 46/46 - 0s - 4ms/step - loss: 6664.9165 - mae: 58.8081 - val_loss: 6203.6729 - val_mae: 56.1273 - learning_rate: 2.5119e-04 Epoch 30/100 46/46 - 0s - 4ms/step - loss: 6666.5435 - mae: 58.8500 - val_loss: 6189.1016 - val_mae: 55.8678 - learning_rate: 2.8184e-04 Epoch 31/100 46/46 - 0s - 4ms/step - loss: 6675.3506 - mae: 58.9821 - val_loss: 6185.7739 - val_mae: 55.6677 - learning_rate: 3.1623e-04 Epoch 32/100 46/46 - 0s - 4ms/step - loss: 6665.8774 - mae: 58.6488 - val_loss: 6200.5791 - val_mae: 56.2484 - learning_rate: 3.5481e-04 Epoch 33/100 46/46 - 0s - 3ms/step - loss: 6671.8237 - mae: 58.8886 - val_loss: 6224.8320 - val_mae: 56.3359 - learning_rate: 3.9811e-04 Epoch 34/100 46/46 - 0s - 4ms/step - loss: 6669.1304 - mae: 58.8822 - val_loss: 6185.3228 - val_mae: 55.7860 - learning_rate: 4.4668e-04 Epoch 35/100 46/46 - 0s - 4ms/step - loss: 6663.7876 - mae: 58.7177 - val_loss: 6186.2969 - val_mae: 55.8232 - learning_rate: 5.0119e-04 Epoch 36/100 46/46 - 0s - 4ms/step - loss: 6659.3828 - mae: 58.6457 - val_loss: 6183.6606 - val_mae: 55.7271 - learning_rate: 5.6234e-04 Epoch 37/100 46/46 - 0s - 4ms/step - loss: 6667.7988 - mae: 58.6138 - val_loss: 6185.0737 - val_mae: 55.7521 - learning_rate: 6.3096e-04 Epoch 38/100 46/46 - 0s - 4ms/step - loss: 6663.3281 - mae: 58.6729 - val_loss: 6186.1538 - val_mae: 55.8061 - learning_rate: 7.0795e-04 Epoch 39/100 46/46 - 0s - 4ms/step - loss: 6661.1553 - mae: 58.7320 - val_loss: 6186.3018 - val_mae: 55.8084 - learning_rate: 7.9433e-04 Epoch 40/100 46/46 - 0s - 4ms/step - loss: 6660.1353 - mae: 58.7124 - val_loss: 6186.2378 - val_mae: 55.8017 - learning_rate: 8.9125e-04 Epoch 41/100 46/46 - 0s - 4ms/step - loss: 6660.4951 - mae: 58.6951 - val_loss: 6185.3828 - val_mae: 55.7754 - learning_rate: 1.0000e-03 Epoch 42/100 46/46 - 0s - 4ms/step - loss: 6660.5625 - mae: 58.6947 - val_loss: 6185.5791 - val_mae: 55.7824 - learning_rate: 0.0011 Epoch 43/100 46/46 - 0s - 3ms/step - loss: 6659.8203 - mae: 58.6846 - val_loss: 6185.5142 - val_mae: 55.7753 - learning_rate: 0.0013 Epoch 44/100 46/46 - 0s - 4ms/step - loss: 6659.0610 - mae: 58.6676 - val_loss: 6185.7871 - val_mae: 55.7912 - learning_rate: 0.0014 Epoch 45/100 46/46 - 0s - 3ms/step - loss: 6659.4551 - mae: 58.6649 - val_loss: 6185.5303 - val_mae: 55.7794 - learning_rate: 0.0016 Epoch 46/100 46/46 - 0s - 4ms/step - loss: 6659.5181 - mae: 58.6710 - val_loss: 6185.6777 - val_mae: 55.7965 - learning_rate: 0.0018 Epoch 47/100 46/46 - 0s - 4ms/step - loss: 6659.9595 - mae: 58.6583 - val_loss: 6185.2598 - val_mae: 55.7675 - learning_rate: 0.0020 Epoch 48/100 46/46 - 0s - 4ms/step - loss: 6659.7246 - mae: 58.6322 - val_loss: 6184.7739 - val_mae: 55.7363 - learning_rate: 0.0022 Epoch 49/100 46/46 - 0s - 3ms/step - loss: 6657.2500 - mae: 58.6726 - val_loss: 6521.6450 - val_mae: 59.5835 - learning_rate: 0.0025 Epoch 50/100 46/46 - 0s - 4ms/step - loss: 6925.8135 - mae: 60.5797 - val_loss: 6185.8862 - val_mae: 55.9682 - learning_rate: 0.0028 Epoch 51/100 46/46 - 0s - 4ms/step - loss: 6668.2227 - mae: 58.8161 - val_loss: 6182.8013 - val_mae: 55.9339 - learning_rate: 0.0032 Epoch 52/100 46/46 - 0s - 4ms/step - loss: 6694.4438 - mae: 58.9545 - val_loss: 6186.1021 - val_mae: 55.7800 - learning_rate: 0.0035 Epoch 53/100 46/46 - 0s - 4ms/step - loss: 6658.6919 - mae: 58.4989 - val_loss: 6185.0527 - val_mae: 55.6605 - learning_rate: 0.0040 Epoch 54/100 46/46 - 0s - 4ms/step - loss: 6663.8101 - mae: 58.6106 - val_loss: 6185.4736 - val_mae: 55.6143 - learning_rate: 0.0045 Epoch 55/100 46/46 - 0s - 3ms/step - loss: 6673.8262 - mae: 58.7408 - val_loss: 6186.1309 - val_mae: 55.7030 - learning_rate: 0.0050 Epoch 56/100 46/46 - 0s - 4ms/step - loss: 6659.2207 - mae: 58.5080 - val_loss: 6184.7388 - val_mae: 55.6649 - learning_rate: 0.0056 Epoch 57/100 46/46 - 0s - 4ms/step - loss: 6659.6392 - mae: 58.5433 - val_loss: 6185.1011 - val_mae: 55.6567 - learning_rate: 0.0063 Epoch 58/100 46/46 - 0s - 4ms/step - loss: 6667.4302 - mae: 58.5852 - val_loss: 6181.9668 - val_mae: 55.7025 - learning_rate: 0.0071 Epoch 59/100 46/46 - 0s - 4ms/step - loss: 6660.6426 - mae: 58.5344 - val_loss: 6185.2515 - val_mae: 55.6256 - learning_rate: 0.0079 Epoch 60/100 46/46 - 0s - 4ms/step - loss: 6659.3555 - mae: 58.5379 - val_loss: 6184.2944 - val_mae: 55.7058 - learning_rate: 0.0089 Epoch 61/100 46/46 - 0s - 4ms/step - loss: 6660.7383 - mae: 58.5535 - val_loss: 6185.3726 - val_mae: 55.6192 - learning_rate: 0.0100 Epoch 62/100 46/46 - 0s - 4ms/step - loss: 6669.2344 - mae: 58.6457 - val_loss: 6185.4141 - val_mae: 55.6365 - learning_rate: 0.0112 Epoch 63/100 46/46 - 0s - 4ms/step - loss: 6745.3027 - mae: 59.4220 - val_loss: 6185.3462 - val_mae: 55.6509 - learning_rate: 0.0126 Epoch 64/100 46/46 - 0s - 4ms/step - loss: 7612.5645 - mae: 61.5971 - val_loss: 6185.1729 - val_mae: 55.6866 - learning_rate: 0.0141 Epoch 65/100 46/46 - 0s - 4ms/step - loss: 6658.2686 - mae: 58.5969 - val_loss: 6185.2148 - val_mae: 55.6957 - learning_rate: 0.0158 Epoch 66/100 46/46 - 0s - 4ms/step - loss: 6658.2686 - mae: 58.6081 - val_loss: 6185.2568 - val_mae: 55.7027 - learning_rate: 0.0178 Epoch 67/100 46/46 - 0s - 4ms/step - loss: 6658.2959 - mae: 58.6179 - val_loss: 6185.3013 - val_mae: 55.7090 - learning_rate: 0.0200 Epoch 68/100 46/46 - 0s - 3ms/step - loss: 6658.3433 - mae: 58.6267 - val_loss: 6185.3462 - val_mae: 55.7146 - learning_rate: 0.0224 Epoch 69/100 46/46 - 0s - 4ms/step - loss: 6658.4072 - mae: 58.6345 - val_loss: 6185.3911 - val_mae: 55.7195 - learning_rate: 0.0251 Epoch 70/100 46/46 - 0s - 4ms/step - loss: 6658.4888 - mae: 58.6413 - val_loss: 6185.4316 - val_mae: 55.7238 - learning_rate: 0.0282 Epoch 71/100 46/46 - 0s - 4ms/step - loss: 6658.5850 - mae: 58.6471 - val_loss: 6185.4673 - val_mae: 55.7273 - learning_rate: 0.0316 Epoch 72/100 46/46 - 0s - 4ms/step - loss: 6658.6953 - mae: 58.6519 - val_loss: 6185.4985 - val_mae: 55.7301 - learning_rate: 0.0355 Epoch 73/100 46/46 - 0s - 4ms/step - loss: 6658.8218 - mae: 58.6558 - val_loss: 6185.5229 - val_mae: 55.7323 - learning_rate: 0.0398 Epoch 74/100 46/46 - 0s - 4ms/step - loss: 6658.9609 - mae: 58.6588 - val_loss: 6185.5415 - val_mae: 55.7339 - learning_rate: 0.0447 Epoch 75/100 46/46 - 0s - 4ms/step - loss: 6659.1143 - mae: 58.6609 - val_loss: 6185.5542 - val_mae: 55.7350 - learning_rate: 0.0501 Epoch 76/100 46/46 - 0s - 4ms/step - loss: 6659.2822 - mae: 58.6624 - val_loss: 6185.5615 - val_mae: 55.7356 - learning_rate: 0.0562 Epoch 77/100 46/46 - 0s - 4ms/step - loss: 6659.4629 - mae: 58.6633 - val_loss: 6185.5654 - val_mae: 55.7359 - learning_rate: 0.0631 Epoch 78/100 46/46 - 0s - 3ms/step - loss: 6659.6577 - mae: 58.6638 - val_loss: 6185.5649 - val_mae: 55.7359 - learning_rate: 0.0708 Epoch 79/100 46/46 - 0s - 4ms/step - loss: 6659.8667 - mae: 58.6639 - val_loss: 6185.5635 - val_mae: 55.7357 - learning_rate: 0.0794 Epoch 80/100 46/46 - 0s - 4ms/step - loss: 6660.0854 - mae: 58.6638 - val_loss: 6185.5591 - val_mae: 55.7353 - learning_rate: 0.0891 Epoch 81/100 46/46 - 0s - 4ms/step - loss: 6660.3091 - mae: 58.6634 - val_loss: 6185.5522 - val_mae: 55.7348 - learning_rate: 0.1000 Epoch 82/100 46/46 - 0s - 4ms/step - loss: 6660.5322 - mae: 58.6629 - val_loss: 6185.5454 - val_mae: 55.7342 - learning_rate: 0.1122 Epoch 83/100 46/46 - 0s - 4ms/step - loss: 6660.7471 - mae: 58.6622 - val_loss: 6185.5361 - val_mae: 55.7334 - learning_rate: 0.1259 Epoch 84/100 46/46 - 0s - 4ms/step - loss: 6660.9487 - mae: 58.6614 - val_loss: 6185.5269 - val_mae: 55.7326 - learning_rate: 0.1413 Epoch 85/100 46/46 - 0s - 4ms/step - loss: 6661.1230 - mae: 58.6606 - val_loss: 6185.5166 - val_mae: 55.7317 - learning_rate: 0.1585 Epoch 86/100 46/46 - 0s - 3ms/step - loss: 6661.2661 - mae: 58.6598 - val_loss: 6185.5088 - val_mae: 55.7310 - learning_rate: 0.1778 Epoch 87/100 46/46 - 0s - 4ms/step - loss: 6661.3799 - mae: 58.6591 - val_loss: 6185.5044 - val_mae: 55.7306 - learning_rate: 0.1995 Epoch 88/100 46/46 - 0s - 4ms/step - loss: 6661.4756 - mae: 58.6590 - val_loss: 6185.5068 - val_mae: 55.7308 - learning_rate: 0.2239 Epoch 89/100 46/46 - 0s - 4ms/step - loss: 6661.5698 - mae: 58.6596 - val_loss: 6185.5181 - val_mae: 55.7319 - learning_rate: 0.2512 Epoch 90/100 46/46 - 0s - 4ms/step - loss: 6661.6919 - mae: 58.6612 - val_loss: 6185.5435 - val_mae: 55.7341 - learning_rate: 0.2818 Epoch 91/100 46/46 - 0s - 3ms/step - loss: 6661.8906 - mae: 58.6640 - val_loss: 6185.5825 - val_mae: 55.7373 - learning_rate: 0.3162 Epoch 92/100 46/46 - 0s - 4ms/step - loss: 6662.2090 - mae: 58.6676 - val_loss: 6185.6309 - val_mae: 55.7411 - learning_rate: 0.3548 Epoch 93/100 46/46 - 0s - 4ms/step - loss: 6662.6807 - mae: 58.6713 - val_loss: 6185.6807 - val_mae: 55.7449 - learning_rate: 0.3981 Epoch 94/100 46/46 - 0s - 4ms/step - loss: 6663.3130 - mae: 58.6743 - val_loss: 6185.7314 - val_mae: 55.7485 - learning_rate: 0.4467 Epoch 95/100 46/46 - 0s - 4ms/step - loss: 6664.1064 - mae: 58.6765 - val_loss: 6185.7783 - val_mae: 55.7518 - learning_rate: 0.5012 Epoch 96/100 46/46 - 0s - 4ms/step - loss: 6665.0068 - mae: 58.6781 - val_loss: 6185.8408 - val_mae: 55.7559 - learning_rate: 0.5623 Epoch 97/100 46/46 - 0s - 4ms/step - loss: 6666.0430 - mae: 58.6848 - val_loss: 6185.9263 - val_mae: 55.7614 - learning_rate: 0.6310 Epoch 98/100 46/46 - 0s - 4ms/step - loss: 6667.2344 - mae: 58.7013 - val_loss: 6186.0479 - val_mae: 55.7686 - learning_rate: 0.7079 Epoch 99/100 46/46 - 0s - 4ms/step - loss: 6668.6074 - mae: 58.7239 - val_loss: 6186.2227 - val_mae: 55.7781 - learning_rate: 0.7943 Epoch 100/100 46/46 - 0s - 4ms/step - loss: 6670.1899 - mae: 58.7511 - val_loss: 6186.4790 - val_mae: 55.7909 - learning_rate: 0.8913
Let's have a look at the graph of the learning rate scheduler. We'll compare it by the loss value of the trained model.
In [ ]:
# create learning rate scheduler graph to find optimal adam optimizer learning rate
lrs = 1e-5 * (10**(np.arange(100)/20)) # recreated formula of exponent, so it aligns in the graph
plt.semilogx(lrs, find_lr_history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding ideal learning rate')
Out[ ]:
Text(0.5, 1.0, 'Finding ideal learning rate')
I've actually ran this exact code from above about 10 times, and every result presents something very different. Out of all the results, it seems to either settle on 10^-3, or 10^1. So let's see how the results look between the two.
MODEL 7 - 0.001 Learning Rate for ADAM¶
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_7 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_7.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
modelwaist_7.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd3d4fcad0>
MODEL 8 - 0.1 Learning Rate for ADAM¶
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_8 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_8.compile(optimizer=Adam(learning_rate=0.1),
loss='mse',
metrics=['mae'])
modelwaist_8.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd402921d0>
In [ ]:
show_scores(modelwaist_7, ywaist_test, Xwaist_test, 'Waist Evaluation'), show_scores(modelwaist_8, ywaist_test, Xwaist_test, 'Waist Evaluation')
WARNING:tensorflow:5 out of the last 13 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x000002BD4A2E20C0> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 21ms/step WARNING:tensorflow:5 out of the last 13 calls to <function TensorFlowTrainer.make_predict_function.<locals>.one_step_on_data_distributed at 0x000002BD4A2E0680> triggered tf.function retracing. Tracing is expensive and the excessive number of tracings could be due to (1) creating @tf.function repeatedly in a loop, (2) passing tensors with different shapes, (3) passing Python objects instead of tensors. For (1), please define your @tf.function outside of the loop. For (2), @tf.function has reduce_retracing=True option that can avoid unnecessary retracing. For (3), please refer to https://www.tensorflow.org/guide/function#controlling_retracing and https://www.tensorflow.org/api_docs/python/tf/function for more details. 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 19ms/step
Out[ ]:
(('Waist Evaluation',
{'Test MAPE (%)': np.float64(97.92764344645435),
'Test MSE': 6315.116589657433,
'Test R^2': -0.021016330090229518}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(99.88777454403663),
'Test MSE': 6185.132946333593,
'Test R^2': -8.158722311080169e-07}))
In [ ]:
ywaist_pred1 = modelwaist_7.predict(Xwaist_test)
ywaist_pred2 = modelwaist_8.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Weight and Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred1, label='Weight prediction 0.001 lr')
ax.plot(ywaist_pred2, label='Waist prediction 0.1 lr')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step
Alright, seems like 0.001 is better than 0.1, as the latter learning rate is way too strong, that it misses the many nuances of the data.
Let's compare the learning rate with the latest model of waist.
In [ ]:
ywaist_pred = modelwaist_6.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
Definitely seems to perform a hell of a lot better with the learning rate of 0.001!
But now that got me to think, should we also see how well other optimizers work on the dataset, rather than relying on Adam? My bias towards Adam, has been due to how it outperforms SGD on practically every data I've worked with. But we need to rule out all possibilities first.
ChatGPT has also suggested to use RMSprop, so we will use this in conjunction with SGD, and also try find their optimal learning rate.
In [ ]:
# create optimal learning rate adam optimizer model
tf.random.set_seed(35)
sgd_model = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
sgd_model.compile(optimizer='sgd',
loss='mse',
metrics=['mae'])
# learning rate scheduler, going through different learning rates in exponential increase, based on epoch number
lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10 ** (epoch/20)) # formula for exponential increase
find_lr_history = sgd_model.fit(Xwaist_train,
ywaist_train,
epochs=100,
verbose=0,
validation_data=(Xwaist_test,ywaist_test),
callbacks=[lr_scheduler])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
# create learning rate scheduler graph to find optimal adam optimizer learning rate
lrs = 1e-5 * (10**(np.arange(100)/20)) # recreated formula of exponent, so it aligns in the graph
plt.semilogx(lrs, find_lr_history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding ideal learning rate')
Out[ ]:
Text(0.5, 1.0, 'Finding ideal learning rate')
In [ ]:
# create optimal learning rate adam optimizer model
tf.random.set_seed(35)
rms_model = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
rms_model.compile(optimizer='rmsprop',
loss='mse',
metrics=['mae'])
# learning rate scheduler, going through different learning rates in exponential increase, based on epoch number
lr_scheduler=tf.keras.callbacks.LearningRateScheduler(lambda epoch: 1e-5 * 10 ** (epoch/20)) # formula for exponential increase
find_lr_history = rms_model.fit(Xwaist_train,
ywaist_train,
epochs=100,
verbose=0,
validation_data=(Xwaist_test,ywaist_test),
callbacks=[lr_scheduler])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
# create learning rate scheduler graph to find optimal adam optimizer learning rate
lrs = 1e-5 * (10**(np.arange(100)/20)) # recreated formula of exponent, so it aligns in the graph
plt.semilogx(lrs, find_lr_history.history['loss'])
plt.xlabel('Learning Rate')
plt.ylabel('Loss')
plt.title('Finding ideal learning rate')
Out[ ]:
Text(0.5, 1.0, 'Finding ideal learning rate')
Hmmm both SGD and RMSprop don't seem to have optimal learning rates. So we'll settle with the default and see how it performs on the data.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_9 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_9.compile(optimizer='sgd',
loss='mse',
metrics=['mae'])
modelwaist_9.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd5c153610>
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_10 = keras.Sequential([
layers.Dense(16, activation='relu', input_shape=(Xweight_train.shape[1],)),
layers.Dense(32, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(256, activation='relu'),
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_10.compile(optimizer='rmsprop',
loss='mse',
metrics=['mae'])
modelwaist_10.fit(Xwaist_train, ywaist_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd5c7517d0>
We will visually compare the three different optimizers using graphs, and see which performs best visually.
In [ ]:
# display the three different optimizers together
sgd_pred = modelwaist_9.predict(Xwaist_test)
rms_pred = modelwaist_10.predict(Xwaist_test)
adam_pred = modelwaist_7.predict(Xwaist_test)
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)
ax[0].set(title='SGD waist prediction', xlabel='Date')
ax[0].plot(sgd_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();
ax[1].set(title='RMSprop waist prediction', xlabel='Date')
ax[1].plot(rms_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();
ax[2].set(title='Adam waist prediction', xlabel='Date')
ax[2].plot(adam_pred, label='Waist prediction')
ax[2].plot(ywaist_test, label='True waist')
ax[2].legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 4ms/step 6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
Wow... SGD just flat out doesn't exist lmao.
With the comparison, Adam (0.001) and RMSprop are actually pretty comparable with each other. Despite RMSprop correctly guessing a positive difference, compared to the negative trough of Adam (0.001), I think I'll stick to Adam, due to the slight variability it has over RMSprop.
Other improvements?¶
So far, the improvements aren't enough to really learn the patterns of the data. They still hover at 0 a lot. Again, I asked ChatGPT for some guidance, to see what else I can add.
Recurrent Neural Networks (RNNs): Examples such as LSTM (Long Short-Term Memory) or GRU (Gated Recurrent Unit), where they capture sequential dependencies. Aka, they look at rolling past data, e.g. the past 7 days, rather than looking at just one day to improve pattern recognition. They are great on time-series data, and can recognize longer-term dependencies/trends.
Other Functions:
- LeakyReLU: Fixes ReLU, by helping it avoid dying neuron issues
- Tanh: Helps if our training data hovers around 0
- Dropout: Prevents overfitting through disabling a few neurons at random, to prevent over-reliance on certain neurons
- Batch Normalization: Stabilizes training, improves gradient flow, and prevents internal covariate shift
MODEL 11 - Prepearing for LSTM¶
To perform LSTM on our data, we'lll need to reshape the data to a 3D array. Where we have (X_data, y_data, window_size) > windows_size is the value for how far the model will view the data from the past, e.g. 7 days in the past.
This helps the model catch short trends over a week, reducing short-term noises.
In [ ]:
# define a function to create a time-series sequence
def create_sequences(X,y,window_size):
Xs, ys = [], []
for i in range(len(X) - window_size):
Xs.append(X[i:i+window_size])
ys.append(y[i+window_size])
return np.array(Xs), np.array(ys)
window_size=7 # size of how far our data will catch trends for (in days)
Xwaist_sequence_train, ywaist_sequence_train = create_sequences(Xwaist_train, ywaist_train, window_size)
Xwaist_sequence_test, ywaist_sequence_test = create_sequences(Xwaist_test, ywaist_test, window_size)
ReLU activation replaced with tanh¶
With the LSTM layers, the activation can't use relu, as they can't squish values between -1 and 1, which can lead to exploding activations and lack of control. So in place is tanh suggested by ChatGPT. It also stated, that ReLU would rather be ideal if it comes after the LSTM layers.
for this model, we'll try LSTM on the first half of increasing layers, and then decrease in ReLU.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_11 = keras.Sequential([
layers.LSTM(16, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(32, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=True),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(256, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_11.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
modelwaist_11.fit(expandX_train, ywaist_sequence_train, epochs=20, verbose=0)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd7b0317d0>
In [ ]:
ywaist_pred = modelwaist_11.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 166ms/step
Wow that was very disappointing...
Maybe we need to change up the neurons and layers of LSTM? ChatGPT has made a model example, where the neuron layer is as follows > 64, 128, 64 for LSTM. So let's try that.
MODEL 12 - Reorganize LSTM layer/neurons¶
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_12 = keras.Sequential([
layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_12.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
modelwaist_12.fit(expandX_train, ywaist_sequence_train, epochs=20, verbose=0)
Out[ ]:
<keras.src.callbacks.history.History at 0x2bd871eed10>
In [ ]:
ywaist_pred = modelwaist_12.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 108ms/step
Again there's nothing... let's have a visual of the ML model's verbose detail, and see if we can troubleshoot it.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_12 = keras.Sequential([
layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_12.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist = modelwaist_12.fit(expandX_train, ywaist_sequence_train, epochs=20, verbose=2)
Epoch 1/20
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
46/46 - 6s - 121ms/step - loss: 6670.5825 - mae: 58.5628 Epoch 2/20 46/46 - 0s - 9ms/step - loss: 6670.1753 - mae: 58.5577 Epoch 3/20 46/46 - 0s - 9ms/step - loss: 6670.1016 - mae: 58.5735 Epoch 4/20 46/46 - 0s - 9ms/step - loss: 6670.0640 - mae: 58.5756 Epoch 5/20 46/46 - 0s - 9ms/step - loss: 6669.9160 - mae: 58.5841 Epoch 6/20 46/46 - 0s - 9ms/step - loss: 6669.8730 - mae: 58.5923 Epoch 7/20 46/46 - 0s - 9ms/step - loss: 6669.8228 - mae: 58.5991 Epoch 8/20 46/46 - 0s - 9ms/step - loss: 6669.7583 - mae: 58.6074 Epoch 9/20 46/46 - 0s - 10ms/step - loss: 6669.7158 - mae: 58.6146 Epoch 10/20 46/46 - 0s - 9ms/step - loss: 6669.6553 - mae: 58.6228 Epoch 11/20 46/46 - 0s - 9ms/step - loss: 6669.6519 - mae: 58.6279 Epoch 12/20 46/46 - 0s - 9ms/step - loss: 6669.5977 - mae: 58.6323 Epoch 13/20 46/46 - 0s - 9ms/step - loss: 6669.5645 - mae: 58.6386 Epoch 14/20 46/46 - 0s - 10ms/step - loss: 6669.5811 - mae: 58.6400 Epoch 15/20 46/46 - 0s - 9ms/step - loss: 6669.4985 - mae: 58.6443 Epoch 16/20 46/46 - 0s - 10ms/step - loss: 6669.4346 - mae: 58.6494 Epoch 17/20 46/46 - 0s - 9ms/step - loss: 6669.4727 - mae: 58.6517 Epoch 18/20 46/46 - 0s - 9ms/step - loss: 6669.4189 - mae: 58.6543 Epoch 19/20 46/46 - 0s - 9ms/step - loss: 6669.3823 - mae: 58.6571 Epoch 20/20 46/46 - 0s - 9ms/step - loss: 6669.3110 - mae: 58.6647
In [ ]:
plt.plot(hist.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
So it seems like Loss is going down, but very slowly. Let's try and train it on 100 epochs to see if that makes a difference.
MODEL 13 - 100 Epochs for LSTM MODEL 12¶
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_13 = keras.Sequential([
layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_13.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_13 = modelwaist_13.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
In [ ]:
plt.plot(hist_13.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
It's improving sure, but it's very marginal. Not going to be efficient if we train this for 1000 epochs, as our resources and time are limited. Let's see how our MODEL 11 with 5 LSTM models do.
MODEL 14 - MODEL 11 has 100 epochs¶
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_14 = keras.Sequential([
layers.LSTM(16, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(32, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=True),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(256, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_14.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_14 = modelwaist_14.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
In [ ]:
plt.plot(hist_14.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Performance is pretty much stalled for the majority of the learning progress... No real progress made until the last 10 epochs...
But what if, the time sequence is too long? We've set it to 7 days, but what if after 7 days of weight and waist difference, the positive gain, is cancelled out with the negative loss? So let's set the time sequence to 3 days, and see how it affects the results with loss.
MODEL 15 - Set Time Sequence for LSTM to 3 days¶
In [ ]:
window_size=3 # changing from 7 to 3
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_15 = keras.Sequential([
layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_15.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_15 = modelwaist_15.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
In [ ]:
plt.plot(hist_15.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Doesn't seem to help with accuracy. Let's try a time series of 14 days instead
MODEL 16 - Time Series extended to 14 instead of 3¶
In [ ]:
window_size=14 # changing from 3 to 14
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_16 = keras.Sequential([
layers.LSTM(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.LSTM(128, activation='tanh', return_sequences=True),
layers.LSTM(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_16.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_16 = modelwaist_16.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
In [ ]:
plt.plot(hist_16.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Seems 7 days is the sweet spot.
MODEL 17 - Switch LSTM for GRU¶
ChatGPT suggested that if LSTM is not performing as expected, that we can try using GRU layers, which does something similar to LSTM, except it is more efficient and works well with smaller datasets like ours.
In [ ]:
window_size=7 # window_size back to 7
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_17 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_17.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_17 = modelwaist_17.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
In [ ]:
plt.plot(hist_17.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Looks like we're finally getting somewhere!
MODEL 18 - Increase GRU layers/neurons¶
Now I want to see what happens if I try increasing the GRU layers, where there's 256 neurons on the biggest GRU layer
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_18 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(256, activation='tanh', return_sequences=True),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='relu'),
layers.Dense(64, activation='relu'),
layers.Dense(32, activation='relu'),
layers.Dense(16, activation='relu'),
layers.Dense(8, activation='relu'),
layers.Dense(1, activation='linear')
])
modelwaist_18.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_18 = modelwaist_18.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
In [ ]:
plt.plot(hist_18.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
So we double the time to train the model, to get essentially no improvement :(
MODEL 19 - Change ReLU to Tanh¶
ChatGPT has stated that Tanh might help with data around zero, so we'll replace ReLU with Tanh and see how it performs.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_19 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='tanh'),
layers.Dense(64, activation='tanh'),
layers.Dense(32, activation='tanh'),
layers.Dense(16, activation='tanh'),
layers.Dense(8, activation='tanh'),
layers.Dense(1, activation='linear')
])
modelwaist_19.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_19 = modelwaist_19.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
In [ ]:
plt.plot(hist_19.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Doesn't work
MODEL 20 - Try LeakyReLU¶
Let's try using LeakyReLU in-place of Tanh (that aren't a part of GRU)
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_20 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
modelwaist_20.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_20 = modelwaist_20.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_20.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Best performance so far!
In [ ]:
ywaist_pred = modelwaist_20.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 140ms/step
The graph so far reflects the risk taking! But we need it to keep doing it for all areas of the data
MODEL 21 - Implement Batch Normalization and Dropout¶
ChatGPT suggests that I try adding Batch Normalization and Dropout into my layers.
Normalization is to help reduce internal covariate shift, making training more stable. They normalize activations by subtracting the mean, and dividing by standard deviation. This is often added after a Dense/Convolutional layer, but before activation functions.
Dropout is to help reduce overfitting by preventing over-reliance to certain neurons. This is done by randomly dropping some neurons to the value of 0, forcing the model to learn patterns without such neuron. It's added after a Dense or LSTM/GRU layer.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_21 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(16),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(8),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(1, activation='linear')
])
modelwaist_21.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_21 = modelwaist_21.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_21.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Didn't work as hoped. Maybe I added it to too many layers?
MODEL 22 - Reduce Batch Normzlization and Dropout layers¶
This time, I'm going to only add Batch Normalization and Dropout layers to 2 GRU layers. The first GRU layer will not have either of these layers.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_22 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
modelwaist_22.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_22 = modelwaist_22.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_22.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
The model seems to have a hard time improving with Batch Normalization and Dropout
Model 23 - Use Batch Normalization and Dropout on LeakyReLU¶
Let's try using the Normalization and Dropout layer on the LeakyReLU layers only
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_23 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(16),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(8),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(1, activation='linear')
])
modelwaist_23.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_23 = modelwaist_23.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_23.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Seems like I'll have to retire Batch Normalization and Dropout from the layers in search of better options.
MODEL 24 - Intermittent switching between LeakyReLU and tanh¶
Since tanh is supposedly good at working with data around 0, I figure that swapping between LeakyReLU layers, and tanh layers would be interesting, and I hope that both their specialty will compliment each other.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_24 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64, activation='tanh'),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16, activation='tanh'),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
modelwaist_24.compile(optimizer=Adam(learning_rate=0.001),
loss='mse',
metrics=['mae'])
hist_24 = modelwaist_24.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_24.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Okay, maybe not then :(
It's back to the drawing board for me to search for solutions to better this ML model. So I've once again asked for guidance with ChatGPT, especially with the model not being risk-taking, and showed the best resulting ML model so far. It has given me suggestions to help it take risks and be less underfitting, so I'll work through each one and see what results we get.
MODEL 25 - Change MSE to Huber Loss, but also finding the ideal delta variable¶
Huber loss is less sensitive to extreme errors, which allows the model to take risks. One extra thing is it has this variable named delta. It acts like a scaled slider between mse and mae. The smaller the delta is, the more it'll become mse. Vice versa for larger delta.
Small delta > less risk taking, Large delta > more risk taking
Problem is, how do we find ideal Delta? ChatGPT has given me a line of code that suggests the ideal Delta. By finding the difference of prediction to truth, getting it's standard deviation, and multiply it by 1.5
In [ ]:
import numpy as np
# Compute residuals
residuals = np.abs(ywaist_sequence_test - ywaist_pred) # Get difference of preds and truth using .abs
# Compute standard deviation
delta = 1.5 * np.std(residuals)
print(f"Ideal Delta: {delta}")
Ideal Delta: 93.23351755665132
Now let's try it on the model. Because we've changed our loss function, and is going to change based on the delta, we'll compare using the values it predicts.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_25 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
modelwaist_25.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_25 = modelwaist_25.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
# display the three different optimizers together
best_pred = modelwaist_20.predict(expandX_test)
huber_pred = modelwaist_25.predict(expandX_test)
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)
ax[0].set(title='Previous Best Prediction', xlabel='Date')
ax[0].plot(best_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();
ax[1].set(title='Huber Delta=93.2 Prediction', xlabel='Date')
ax[1].plot(huber_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();
ax[2].set(title='Loss over Epoch', xlabel='Epoch', ylabel='Loss')
ax[2].plot(hist_25.history['loss'])
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step 5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
Out[ ]:
[<matplotlib.lines.Line2D at 0x2be2f5f2750>]
Hmmm no extreme risk taking, might need to increase delta
MODEL 26 - Delta set to 1000¶
Might as well try the most extreme of extreme. If a Delta of 93.2 is not doing much, we might as well try an extreme value, and see how the ML model reacts to it.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_26 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
modelwaist_26.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=1000),
metrics=['mae'])
hist_26 = modelwaist_26.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
# display the three different optimizers together
best_pred = modelwaist_20.predict(expandX_test)
huber_pred = modelwaist_26.predict(expandX_test)
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)
ax[0].set(title='Previous Best Prediction', xlabel='Date')
ax[0].plot(best_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();
ax[1].set(title='Huber Delta=1000 Prediction', xlabel='Date')
ax[1].plot(huber_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();
ax[2].set(title='Loss over Epoch', xlabel='Epoch', ylabel='Loss')
ax[2].plot(hist_26.history['loss'])
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 11ms/step 5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step
Out[ ]:
[<matplotlib.lines.Line2D at 0x2be4536bb50>]
Uh oh, seems to do way worse, despite the higher the value, the more risk taking it does!
MODEL 27 - Delta GridSearchCV¶
One other thing ChatGPT suggested, is by using GridSearchCV, with multiple different delta values. And then having it return the gest delta value, based on the presets we've given to the model.
I've set the GridSearch upon the best model so far, model_20 and will see what happens.
In [ ]:
from tensorflow.keras.losses import Huber
from tensorflow.keras.optimizers import Adam
deltas = [1, 5, 10, 20, 50]
best_delta = None
best_val_loss = float("inf")
for delta in deltas:
modelwaist_20.compile(optimizer=Adam(learning_rate=0.001), loss=Huber(delta=delta), metrics=['mae'])
history = modelwaist_20.fit(expandX_train, ywaist_sequence_train, validation_data=(expandX_test, ywaist_sequence_test), epochs=50, verbose=0)
val_loss = min(history.history['val_loss']) # Get best validation loss
if val_loss < best_val_loss:
best_val_loss = val_loss
best_delta = delta
print(f"Optimal Delta: {best_delta}")
Optimal Delta: 1
Surprising it gave 1 as the best delta value, despite smaller value mean less risk taking.
Regardless, let's try it on model
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_27 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
modelwaist_27.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=1000),
metrics=['mae'])
hist_27 = modelwaist_27.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
# display the three different optimizers together
best_pred = modelwaist_20.predict(expandX_test)
huber_pred = modelwaist_27.predict(expandX_test)
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)
ax[0].set(title='Previous Best Prediction', xlabel='Date')
ax[0].plot(best_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();
ax[1].set(title='Huber Delta=1 Prediction', xlabel='Date')
ax[1].plot(huber_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();
ax[2].set(title='Loss over Epoch', xlabel='Epoch', ylabel='Loss')
ax[2].plot(hist_27.history['loss'])
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step 5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 10ms/step
Out[ ]:
[<matplotlib.lines.Line2D at 0x2be4abbc610>]
Aah I see why that's best performing, because without taking risks and staying at 0, you are closest to all predictions, over risking wrong predictions!
MODEL 28 - Reduce Adam Learning Rate Overtime¶
ChatGPT theorized that, the learning rate for Adam, is making the model stuck in doing small changes, and settling too ealy near zero. Reducing learning rate overtime, helps it fine-tune its predictions, rather than getting stuck on bad local minimums.
We'll pull in MODEL_25 and try our learning rate reducing experiment on it.
In [ ]:
from tensorflow.keras.callbacks import ReduceLROnPlateau
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-5) # Do not reduce below this
# Include in model training
hist_28 = modelwaist_25.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])
Using the MODEL_25, means we can compare loss meaningfully. Based on the graph, we need something lower than 2757 loss to show that there's improvement.
In [ ]:
plt.plot(hist_28.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Results! What if we made the minimum learning rate, even smaller in value?
MODEL_29 - min_lr becomes 1e-10, and factor becomes 0.25¶
Again, trying some extreme values to see what happens
In [ ]:
from tensorflow.keras.callbacks import ReduceLROnPlateau
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.25, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
# Include in model training
hist_29 = modelwaist_25.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_29.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Hmm it started with a much lower loss compared to MODEL_28, but just doesn't improve anymore.
In [ ]:
ywaist_pred = modelwaist_25.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 12ms/step
Seems we're encountering a pattern, where the model is only predicting at the same are of the graph... However, it is making the kind of risk taking prediction that I've been looking for!
Let's try one more thing before we dive in deeper.
MODEL_30 - Use Swish over LeakyReLU / decreased min_lr¶
Last thing ChatGPT suggested, is swapping LeakyReLU for swish. What it does, it allows small negative values, whereas ReLU outputs zero on negative inputs. This can block gradient flow. Swish smoothly transitions from negative to positive values, works better in deeper models with gradient flow improvement, and has a value named x in the equation, to allow the learning of when to supress or amplify values.
I've decided to further decrease the minimum learning rate, to see what happens
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_30 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128, activation='swish'),
layers.Dense(64, activation='swish'),
layers.Dense(32, activation='swish'),
layers.Dense(16, activation='swish'),
layers.Dense(8, activation='swish'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.25, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-20) # Do not reduce below this
modelwaist_30.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_30 = modelwaist_30.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs)
In [ ]:
plt.plot(hist_30.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
In [ ]:
ywaist_pred = modelwaist_30.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 152ms/step
Not too impressive, but it seems to have the same issue with most if not all previous model. All predictions is fitting within the area between index 10 to index 40 on the x-axis!
MODEL_31 - reintroduce Normalizaton and Dropout¶
Again ChatGPT suggest we solve the issue with Batch Normalization and Dropout. This time, I've reduced the number of these layers, and only added them, after the 3 GRU layers.
Instead of factor=0.25 in Learning Rate Scheduler, I've set it back to 0.5, as it did show promising results of continually decreasing loss values.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_31 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_31.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_31 = modelwaist_31.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_31.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Getting some slightly better results!
MODEL_32 - Add more Batch Normalization and Dropout¶
With further experimentation, I'm just going to add 2 more Batch Normalization and Dropout in the layer as I see fit, and see the resulting outcome.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_32 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_32.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_32 = modelwaist_32.fit(expandX_train, ywaist_sequence_train, epochs=100, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_32.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Not good results
MODEL_33 - Epochs increase to 1,000¶
One big hail mary, is to increase the epoch. I notice how a lot of the graphs show how Loss seems to further decrease at a constant rate the more epochs there is. I'm hesitant in increasing epochs, as it's one of the easiest way to increase training time. However I'm close to exhausting all my options, so I might as well try this one time and see the results.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_33 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_33.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_33 = modelwaist_33.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_33.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
We can see how loss continuoually decreases linearly, and reaches a point where it stops improving. Likely because Learning Rate of Adam becomes too small.
In [ ]:
ywaist_pred = modelwaist_33.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 127ms/step
Finally! We're seeing predictions outside the index 10 to index 40 area!
The likely reason why there are 0 predictions, could be because the ML model didn't have enough time to train and familiarize in all aspects of the data. It probably only recognize the most prominent parts of the data. Aka, the difference value that's most extreme (around index 10 to index 40). Only has it been trained even more, can it effectively predice smaller difference value.
MODEL_34 - Increase factor from 0.5 to 0.8¶
I'll increase factor. Due to our loss graph, we are decreasing the Learning Rate too fast, that it flattens out and doesn't learn anything meaningful. So increasing it allows more time for higher Learning Rate epochs to make a dent on prediction.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_34 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.8, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_34.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_34 = modelwaist_34.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_34.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
We're getting even better results!
This time, it's flattening around epoch 400
In [ ]:
ywaist_pred = modelwaist_34.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 1s 129ms/step
Again, the more the model learns, the more likely the model starts to make predictions!
MODEL_35 - factor 0.8 to 0.92¶
One thing I wanted to figure out the cutoff for where the loss starts to flatten out. We know Learning Rate starts at 0.001, and every 10 epochs, it reduces by 80%. But we want to find the Learning Rate at Epoch 400, where Loss flattens.
This comes out to the following equation:
$\\ \text{lr}_n = \text{lr}_0 \times (0.8)^{\frac{n}{10}} $
$\\ \text{lr}_{400} = 0.001 \times (0.8)^{\frac{400}{10}} = 0.001 \times (0.8)^{40} \approx 1.33 \times 10^{-7} $
1.33 * 10^-7 is the approximate Learning Rate where learning flattens. Let's extrapolate it, and set that at Epoch 1000, rather than on Epoch 400.
$\\ \ 1.33 \times 10^{-7} = 0.001 \times (r)^{\frac{1000}{10}} -> r = \left(\frac{1.33 \times 10^{-7}}{\text{0.001}}\right)^{\frac{10}{1000}} $
Based on the equation and rearrangement, the factor is around 0.9146, or reducing by 8.54% every 10 epochs. Based on that, I'll try setting the factor at 0.92, a little higher than suggested so reduceis slower.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_35 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.92, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_35.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_35 = modelwaist_35.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) Cell In[419], line 45 36 reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss') 37 factor=0.92, # Reduce LR by half when triggered 38 patience=10, # Wait 10 epochs before reducing 39 min_lr=1e-10) # Do not reduce below this 41 modelwaist_35.compile(optimizer=Adam(learning_rate=0.001), 42 loss=Huber(delta=93.2), 43 metrics=['mae']) ---> 45 hist_35 = modelwaist_35.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr]) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py:117, in filter_traceback.<locals>.error_handler(*args, **kwargs) 115 filtered_tb = None 116 try: --> 117 return fn(*args, **kwargs) 118 except Exception as e: 119 filtered_tb = _process_traceback_frames(e.__traceback__) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py:371, in TensorFlowTrainer.fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq) 369 for step, iterator in epoch_iterator: 370 callbacks.on_train_batch_begin(step) --> 371 logs = self.train_function(iterator) 372 callbacks.on_train_batch_end(step, logs) 373 if self.stop_training: File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py:219, in TensorFlowTrainer._make_function.<locals>.function(iterator) 215 def function(iterator): 216 if isinstance( 217 iterator, (tf.data.Iterator, tf.distribute.DistributedIterator) 218 ): --> 219 opt_outputs = multi_step_on_iterator(iterator) 220 if not opt_outputs.has_value(): 221 raise StopIteration File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\util\traceback_utils.py:150, in filter_traceback.<locals>.error_handler(*args, **kwargs) 148 filtered_tb = None 149 try: --> 150 return fn(*args, **kwargs) 151 except Exception as e: 152 filtered_tb = _process_traceback_frames(e.__traceback__) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\polymorphic_function.py:833, in Function.__call__(self, *args, **kwds) 830 compiler = "xla" if self._jit_compile else "nonXla" 832 with OptionalXlaContext(self._jit_compile): --> 833 result = self._call(*args, **kwds) 835 new_tracing_count = self.experimental_get_tracing_count() 836 without_tracing = (tracing_count == new_tracing_count) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\polymorphic_function.py:878, in Function._call(self, *args, **kwds) 875 self._lock.release() 876 # In this case we have not created variables on the first call. So we can 877 # run the first trace but we should fail if variables are created. --> 878 results = tracing_compilation.call_function( 879 args, kwds, self._variable_creation_config 880 ) 881 if self._created_variables: 882 raise ValueError("Creating variables on a non-first call to a function" 883 " decorated with tf.function.") File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\tracing_compilation.py:139, in call_function(args, kwargs, tracing_options) 137 bound_args = function.function_type.bind(*args, **kwargs) 138 flat_inputs = function.function_type.unpack_inputs(bound_args) --> 139 return function._call_flat( # pylint: disable=protected-access 140 flat_inputs, captured_inputs=function.captured_inputs 141 ) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\concrete_function.py:1322, in ConcreteFunction._call_flat(self, tensor_inputs, captured_inputs) 1318 possible_gradient_type = gradients_util.PossibleTapeGradientTypes(args) 1319 if (possible_gradient_type == gradients_util.POSSIBLE_GRADIENT_TYPES_NONE 1320 and executing_eagerly): 1321 # No tape is watching; skip to running the function. -> 1322 return self._inference_function.call_preflattened(args) 1323 forward_backward = self._select_forward_and_backward_functions( 1324 args, 1325 possible_gradient_type, 1326 executing_eagerly) 1327 forward_function, args_with_tangents = forward_backward.forward() File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\atomic_function.py:216, in AtomicFunction.call_preflattened(self, args) 214 def call_preflattened(self, args: Sequence[core.Tensor]) -> Any: 215 """Calls with flattened tensor inputs and returns the structured output.""" --> 216 flat_outputs = self.call_flat(*args) 217 return self.function_type.pack_output(flat_outputs) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\polymorphic_function\atomic_function.py:251, in AtomicFunction.call_flat(self, *args) 249 with record.stop_recording(): 250 if self._bound_context.executing_eagerly(): --> 251 outputs = self._bound_context.call_function( 252 self.name, 253 list(args), 254 len(self.function_type.flat_outputs), 255 ) 256 else: 257 outputs = make_call_op_in_graph( 258 self, 259 list(args), 260 self._bound_context.function_call_options.as_attrs(), 261 ) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\context.py:1683, in Context.call_function(self, name, tensor_inputs, num_outputs) 1681 cancellation_context = cancellation.context() 1682 if cancellation_context is None: -> 1683 outputs = execute.execute( 1684 name.decode("utf-8"), 1685 num_outputs=num_outputs, 1686 inputs=tensor_inputs, 1687 attrs=attrs, 1688 ctx=self, 1689 ) 1690 else: 1691 outputs = execute.execute_with_cancellation( 1692 name.decode("utf-8"), 1693 num_outputs=num_outputs, (...) 1697 cancellation_manager=cancellation_context, 1698 ) File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 51 try: 52 ctx.ensure_initialized() ---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 54 inputs, attrs, num_outputs) 55 except core._NotOkStatusException as e: 56 if name is not None: KeyboardInterrupt:
In [ ]:
plt.plot(hist_35.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Not what I expected. Despite slower reduction, Loss didn't reduce as I thought. Maybe because once it reaches to a certain amount of epochs, it stops learning meaningful information?
MODEL_36 - Change Factor from 0.92 to 0.9¶
Lowering Factor just by a tiny bit, to see what the result is in Loss. See if my hypothesis of learning less meaning info after a certain amount of epochs.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_36 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.9, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_36.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_36 = modelwaist_36.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_36.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Seems like our hypothesis is somewhat correct. The range of epoch where it starts to flatten sits from 400 to 600.
What's good is the lower of Factor also improved Loss value.
So now we know to cut Epochs to 600, and save time in model training, and the range of optimal factor is between 0.8 to 0.9.
MODEL_37 - Set Epoch to 600, and factor to 0.85¶
Epoch to 600, based on previous training results, where improvement flattens around 400 to 600
Factor to 0.85, as we tried 0.8 and 0.9, assuming the ideal value is between these two numbers, based on the resulting Loss graphs.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_37 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.BatchNormalization(),
layers.Dropout(0.2),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(32),
layers.LeakyReLU(alpha=0.1),
layers.Dense(16),
layers.LeakyReLU(alpha=0.1),
layers.Dense(8),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.85, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_37.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_37 = modelwaist_37.fit(expandX_train, ywaist_sequence_train, epochs=600, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_37.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Further tests:¶
Now training each singular model is taking so much time, and takes up so much space through this notebook, I figured that I just run through many different options, and make a list of the changes I've made and what happened.
Factor - 0.7 > 0.6 > 0.75: There has no significant improvement when changing the factor
Delta - 150: I thought about changing the delta, and see if it could improve model. No improvement
Lr 0.01: I thought that, maybe the model is reaching a limit in the learning rate and its reaching too quickly, causing it to flatten. So I made the learning rate higher to make it reach that boundary slower. Didn't work
Patience 20: On the similar vein of Lr 0.01, where we slow down reaching the learning rate, that flattens training rate. No improvement
Epoch 200: I wanted to see what happens if epoch was cut down to a point before the flattening. Results showed, that the model still flattened earlier than the last 200 epochs
Return back to best model: I wanted to see if the loss is still the same with the same model. Surprisingly, no. It flattened out earlier, and did not improve at the same rate.
There's something wrong... Again returned to chatgpt for some answers. It states that the model is getting stuck at a local minimum, and plateauing at ranomd epochs. It can suggest that training dynamics are unstable.
So out of experimentation reasoning, I'll have chatgpt recreate the model with what it thinks will improve the flattening
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
from tensorflow.keras.callbacks import ReduceLROnPlateau
tf.random.set_seed(35)
expandX_train = Xwaist_sequence_train # No np.expand_dims
expandX_test = Xwaist_sequence_test
# Learning rate schedule
initial_lr = 0.001
lr_schedule = keras.optimizers.schedules.ExponentialDecay(
initial_lr, decay_steps=200, decay_rate=0.96, staircase=True
)
# Create model
modelwaist_38 = keras.Sequential([
layers.Bidirectional(layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2]))),
layers.Bidirectional(layers.GRU(128, activation='tanh', return_sequences=True)),
layers.Bidirectional(layers.GRU(64, activation='tanh', return_sequences=False)),
layers.Dropout(0.3),
layers.BatchNormalization(),
layers.Dense(128, activation='swish'),
layers.Dense(64, activation='swish'),
layers.Dense(32, activation='swish'),
layers.Dense(16, activation='swish'),
layers.Dense(8, activation='swish'),
layers.Dense(1, activation='linear')
])
optimizer = keras.optimizers.SGD(learning_rate=0.01, momentum=0.9)
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=20, restore_best_weights=True)
reduce_lr = ReduceLROnPlateau(monitor='loss',
factor=0.5,
patience=5,
min_lr=1e-6)
modelwaist_38.compile(optimizer=optimizer,
loss=Huber(delta=50), # Adjusted delta
metrics=['mae'])
hist_38 = modelwaist_38.fit(expandX_train, ywaist_sequence_train,
epochs=1000,
verbose=0,
callbacks=[reduce_lr, early_stop])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\callbacks\early_stopping.py:153: UserWarning: Early stopping conditioned on metric `val_loss` which is not available. Available metrics are: loss,mae,learning_rate current = self.get_monitor_value(logs)
Changes¶
- Change batch normalization and dropout order - The order of these two layers were not optimal to improvement, so they've been switched
- Overfitting or underfitting - To prevent that, we add early stopping when model stagnates or shows no improvements
- Trying different optimizers - Changing Adam to SGD, might work better
In [ ]:
plt.plot(hist_38.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
In [ ]:
ywaist_pred = modelwaist_38.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 2s 285ms/step
Didn't even work
CHAT GPT MODEL¶
I want to see what ChatGPT generates for a model, and see how different it is from my best performing model, and whether it would be better in prediction as well.
In [ ]:
import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.callbacks import ReduceLROnPlateau
from tensorflow.keras.losses import Huber
# Load or preprocess your dataset
# expandX_train, ywaist_sequence_train = ...
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# Define the model
gptmodel = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dropout(0.2),
layers.Dense(64, activation=None),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dropout(0.2),
layers.Dense(32, activation=None),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dense(1) # Output layer for regression
])
# Define optimizer with a fixed learning rate
optimizer = keras.optimizers.Adam(learning_rate=0.0005)
# Learning rate scheduler
reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.5, patience=50, min_lr=1e-6)
# Compile the model
gptmodel.compile(optimizer=optimizer, loss=Huber(delta=50), metrics=['mae'])
# Train the model
gpthistory = gptmodel.fit(expandX_train, ywaist_sequence_train,
epochs=1000, verbose=1, callbacks=[reduce_lr])
In [ ]:
plt.plot(gpthistory.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Bingo! Finally the flattening loss has been solved. Let's see how well it performs on the test data as well.
In [ ]:
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1) # (None, 210)
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1) # (None, 210)
ywaist_pred = gptmodel.predict(expandX_test)
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 31ms/step
In [ ]:
ywaist_pred = gptmodel.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 8ms/step
Seems like it has finally done what I've been looking for this whole time!
The next step for me is to try and understand what made this model, so much better than my own fabricated one?
Difference between GPT Model to My Model¶
3D Layer Redundant: GPT has chosen to remove the 3 dimensional layer, and keep it as it was (2 dimensional). But why is that?
GRU Layer gone: The main purpose for 3D layer was for GRU. The 3rd dimension was used for sequential data, to keep a range of days before the current day. This helps GRU look at previous days data to make a prediction.
Batch Normalization After Dense Layer: GPT's strategy is putting Normalization layers after a Dense layer of neurons.
Dropout After Activations: GPT's strategy is putting the Dropout layers after an activation function, such as ReLU or Tanh. This is not done on the last activation, as right after is a 1 neuron Dense layer, which won't do any pattern recognition.
Activations changed: LeakyReLU has been substituted for ReLU.
Reduced Activation Layers: GPT has heavily reduced the layers in the model, keeping just 3 ReLU layers, from 128, to 64, 32 neurons.
Adam Learning Rate Hyperparameter at 0.0005: GPT chosen an even smaller learning rate.
Reduce LR Hyperparameter Change: Factor = 0.5, Patience = 50, min_lr = 1e-6. What's different from ours, is Learning Rate is reduced faster, at 50%, however it's reduced even slower, by every 50 epochs, while increasing the minimum Learning Rate from ^-10, to ^-6.
Huber Hyperparameter change: Delta is changed to 50, from 93.2
What point will improve most?¶
There's a good number of changes, but what would yield the most improvement to our current model? Based on what I've experimented so far, the ones that are based on hyperparameter change, will have minimal change in effect.
What would be most contentious, is the GRU, and Normalization/Dropout layer order. GRU essentially changes the entire training data to fit itself, and the way you organize Normalization and Dropout can significantly change the data that the other layers recieve. Plus, ChatGPT has srutinized how my Nomralization/Dropout layers are not placed optimally. The ReLU layers along with the simplicity might play a role as well.
Next step:¶
I'm going to now experiment with the GRU layers, and Normalization/Dropout layers with my current model, in the hopes that from what I've learnt with the GPTmodel, I can create an even more accurate model, and avoid loss flattening. If that doesn't work, then I'll look at minimizing layers as well.
MODEL_39 - Refit Normalization and Dropout¶
I'm going to copy the way ChatGPT organized and places its Batch Normalization and Dropout layers.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_39 = keras.Sequential([
layers.GRU(64, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.GRU(128, activation='tanh', return_sequences=True),
layers.GRU(64, activation='tanh', return_sequences=False), # Last LSTM layer should have return_sequences=False
layers.Dense(128),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(16),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(8),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.8, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_39.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_39 = modelwaist_39.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_39.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
It doesn't seem to make much of an improvement
MODEL_40 - Remove GRU layers¶
Remove the 3 GRU layers on the top. Since these layers use 3 dimensional training data to learn, we will also need to remove the expansion as well, and return it back to 2 dimensional for TensorFlow.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_40 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(16),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(8),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.8, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_40.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_40 = modelwaist_40.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_40.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Seems like strong correlation that having RNN layers like GRU is what's gatekeeping the model from making any real progression/learning.
Definitely feel pretty shitty how much time was spent to perfect the model using GRU, but now we know to keep it on ReLu or something related.
MODEL_41 - Replace LeakyReLU with ReLU¶
Further copying ChatGPT and see if the change in activations will make a difference.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_41 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dropout(0.2),
layers.Dense(16),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dropout(0.2),
layers.Dense(8),
layers.BatchNormalization(),
layers.ReLU(),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.8, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_41.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_41 = modelwaist_41.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_41.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Seems like LeakyReLu and ReLU are quite easily matched! Since ChatGPT stated that LeakyReLU is a bit better than ReLU, we'll stick to it.
MODEL_42 - Removed 16 and 8 neuron LeakyReLU layers¶
Copy the layer layout and removing 16 and 8 neuron layers. Having so many small neuron layers, might make it hard for the model to give varied predictions, as it's limited to a few options only.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_42 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.8, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_42.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_42 = modelwaist_42.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_42.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Yep that seem's to make it better a bit.
MODEL_43 - Reinsert one GRU layer¶
Just to be extra sure of my current conclusion, I want to try one GRU layer, and see if it really creates the previous issue we had, which is flattening in performance.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = np.expand_dims(Xwaist_sequence_train, axis=-1) # Expanding last axis for feature dimension
expandX_test = np.expand_dims(Xwaist_sequence_test, axis=-1)
# create model including relu and linear
modelwaist_43 = keras.Sequential([
layers.GRU(256, activation='tanh', return_sequences=True, input_shape=(expandX_train.shape[1], expandX_train.shape[2])),
layers.Dense(128),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.8, # Reduce LR by half when triggered
patience=10, # Wait 10 epochs before reducing
min_lr=1e-10) # Do not reduce below this
modelwaist_43.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_43 = modelwaist_43.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\rnn\rnn.py:200: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_43.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Our conclusion is proven correct again.
MODEL_44 - Set reduce_lr to ChatGPT's model¶
Changed hyperparameter to match the ones suggested by ChatGPT's model
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_44 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=50, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_44.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=93.2),
metrics=['mae'])
hist_44 = modelwaist_44.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_44.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Accuracy up! Seems the lowering of learning rate could do a little slowing down, to let the model learn on the most optimal learning rate for as long as possible.
MODEL_45 - learning_rate=0.0005, delta=50¶
Further copying ChatGPT's suggested model
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_45 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=50, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_45.compile(optimizer=Adam(learning_rate=0.0005),
loss=Huber(delta=50),
metrics=['mae'])
hist_45 = modelwaist_45.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_45.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
More progress made
MODE_46 - Patience 50>80, delta 50>25¶
Increased patience, to allow the reduction of learning rate to slow more. I'm hoping this will keep the model on the optimal learning rates for as long as possible, and prevent it from quickly flattening.
Delta was changed, because it showed improvement when it was lowered from 93.2 to 50. I wonder what 25 would do?
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_46 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=80, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_46.compile(optimizer=Adam(learning_rate=0.0005),
loss=Huber(delta=25),
metrics=['mae'])
hist_46 = modelwaist_46.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_46.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
MODEL_47 - Change LeakyReLU to Tanh¶
ChatGPT had stated before that Tanh worked well on data that's close to 0. I was wondering whether GRU layers were holding it back performance-wise, so I want to try it out when replacing LeakyReLU
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model with tanh
modelwaist_47 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=80, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_47.compile(optimizer=Adam(learning_rate=0.0005),
loss=Huber(delta=25),
metrics=['mae'])
hist_47 = modelwaist_47.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
plt.plot(hist_47.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Tanh works better than LeakyReLU, and so we'll use it
MODEL_48 - patience 80>100, delta 25>10, learning_rate 0.0005>0.0001¶
Our previous changes of patience (higher) delta (lower) and learning rate (lower), has proven fruitful in lowering loss, and we'll continue to push the limits of these values altogether
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model with tanh
modelwaist_48 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=100, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_48.compile(optimizer=Adam(learning_rate=0.0001),
loss=Huber(delta=10),
metrics=['mae'])
hist_48 = modelwaist_48.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_48.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Hold up... I just realized that Loss is minimizing from it's starting point...
I just remembered that changing delta changes the weight between mae and mse. I want to double check if that's the case by visualizing graph prediction.
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
ywaist_pred = modelwaist_48.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 41ms/step
So we need to find a way to compare model improvement, when I change up delta. What we can do, is calculate the ratio between starting loss, and ending loss after 1000 epochs. We'll make a rough estimate on what it starts and ends.
MODEL_44 : 2800 > 350 = 8
MODEL_45 : 2000 > 350 = 5.71
MODEL_46 : 1200 > 300 = 4
MODEL_47 : 1200 > 250 = 4.8
MODEL_48 : 550 > 300 = 1.833
Based on the new model comparison, MODEL_44 theoretically should be the best performing model. What we'll do is compare the three model with the highest ratio.
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
# display the three different optimizers together
model44_pred = modelwaist_44.predict(expandX_test)
model45_pred = modelwaist_45.predict(expandX_test)
model47_pred = modelwaist_47.predict(expandX_test)
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)
ax[0].set(title='MODEL_44 prediction', xlabel='Date')
ax[0].plot(model44_pred, label='Waist prediction')
ax[0].plot(ywaist_test, label='True waist')
ax[0].legend();
ax[1].set(title='MODEL_45 prediction', xlabel='Date')
ax[1].plot(model45_pred, label='Waist prediction')
ax[1].plot(ywaist_test, label='True waist')
ax[1].legend();
ax[2].set(title='MODEL_47 prediction', xlabel='Date')
ax[2].plot(model47_pred, label='Waist prediction')
ax[2].plot(ywaist_test, label='True waist')
ax[2].legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 40ms/step 5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 49ms/step 5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
The hypothesis seems to match somewhat, so we'll continue improving MODEL_44
MODEL_49 - Delta = 150¶
Set Delta to 150. Seems like the lower the Delta is, the smaller the ratio is from start to end loss value
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_49 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=50, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_49.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=150),
metrics=['mae'])
hist_49 = modelwaist_49.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_49.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
ywaist_pred = modelwaist_49.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
Perfect! The higher the Delta is, the more the model takes risks to predict the Waist Difference!
MODEL_50 - Delta 150 > 250, LeakyReLU > Tanh, Patience 50 > 80¶
Push Delta higher and see if the model continues to push it's risk taking. LeakyReLU to Tanh as it proved successful in another model. Patience 50 saw too quick of a drop off in loss graph, and Patience 100 still had more opportunity to continue decreasing loss. Patience 80 is me trying to find a middle ground between the two.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_50 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=80, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_50.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=250),
metrics=['mae'])
hist_50 = modelwaist_50.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_50.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
Both graph look quite similar, so it's hard to tell which is better. I remember making a function to show scores of my models, so we'll use that
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
show_scores(modelwaist_49, ywaist_test, expandX_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, expandX_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
--------------------------------------------------------------------------- ValueError Traceback (most recent call last) Cell In[521], line 3 1 expandX_test = expandX_test.reshape(expandX_test.shape[0], -1) ----> 3 show_scores(modelwaist_49, ywaist_test, expandX_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, expandX_test, 'Waist Evaluation') Cell In[165], line 36, in show_scores(model, truth, test, title) 32 preds = model.predict(test) 34 # calculating scores 35 scores = { ---> 36 "Test MAPE (%)":mean_absolute_percent_error(truth, preds), 37 "Test MSE":mean_squared_error(truth, preds), 38 "Test R^2":r2_score(truth,preds) # model scoring for r^2, done by doing prediction to X, to y 39 } 41 return title, scores Cell In[165], line 12, in mean_absolute_percent_error(y_true, y_pred) 9 y_pred = np.squeeze(y_pred) 11 # Task1 > If true is 0, and pred is not 0, set to 100% ---> 12 percent_error[(y_true == 0) & (y_pred != 0)] = 100 14 # Task2 > If true and pred is 0, set to 0% 15 percent_error[(y_true == 0) & (y_pred == 0)] = 0 ValueError: operands could not be broadcast together with shapes (164,) (150,)
Seems like our prediction and test data shape are different from each other!
In [ ]:
print(expandX_test.shape[0], ywaist_test.shape[0])
Indeed it is when printing it's shape out. There must be something we did in the past that have altered the data shape, and it's likely the moving average/data window.
In [ ]:
# define a function to create a time-series sequence
def create_sequences(X,y,window_size):
Xs, ys = [], []
for i in range(len(X) - window_size):
Xs.append(X[i:i+window_size])
ys.append(y[i+window_size])
return np.array(Xs), np.array(ys)
window_size=7 # size of how far our data will catch trends for (in days)
Xwaist_sequence_train, ywaist_sequence_train = create_sequences(Xwaist_train, ywaist_train, window_size)
Xwaist_sequence_test, ywaist_sequence_test = create_sequences(Xwaist_test, ywaist_test, window_size)
This is likely the original culprit, and we'll try retrain model_49 and model_50 with Xwaist_train/test.
Refit MODEL_49 and MODEL_50 on original training/testing data¶
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_50 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(Xwaist_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=80, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_50.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=250),
metrics=['mae'])
hist_50 = modelwaist_50.fit(Xwaist_train, ywaist_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
# create model including relu and linear
modelwaist_49 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(Xwaist_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=50, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_49.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=150),
metrics=['mae'])
hist_49 = modelwaist_49.fit(Xwaist_train, ywaist_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
show_scores(modelwaist_49, ywaist_test, Xwaist_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, Xwaist_test, 'Waist Evaluation')
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
--------------------------------------------------------------------------- InvalidArgumentError Traceback (most recent call last) Cell In[558], line 1 ----> 1 show_scores(modelwaist_49, ywaist_test, Xwaist_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_test, Xwaist_test, 'Waist Evaluation') Cell In[165], line 32, in show_scores(model, truth, test, title) 29 def show_scores(model, truth, test, title): 30 31 # model predicting values from training ---> 32 preds = model.predict(test) 34 # calculating scores 35 scores = { 36 "Test MAPE (%)":mean_absolute_percent_error(truth, preds), 37 "Test MSE":mean_squared_error(truth, preds), 38 "Test R^2":r2_score(truth,preds) # model scoring for r^2, done by doing prediction to X, to y 39 } File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py:122, in filter_traceback.<locals>.error_handler(*args, **kwargs) 119 filtered_tb = _process_traceback_frames(e.__traceback__) 120 # To get the full stack trace, call: 121 # `keras.config.disable_traceback_filtering()` --> 122 raise e.with_traceback(filtered_tb) from None 123 finally: 124 del filtered_tb File ~\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tensorflow\python\eager\execute.py:53, in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name) 51 try: 52 ctx.ensure_initialized() ---> 53 tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name, 54 inputs, attrs, num_outputs) 55 except core._NotOkStatusException as e: 56 if name is not None: InvalidArgumentError: Graph execution error: Detected at node sequential_151_1/dense_935_1/BiasAdd defined at (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel_launcher.py", line 18, in <module> File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\traitlets\config\application.py", line 1075, in launch_instance File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelapp.py", line 739, in start File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\tornado\platform\asyncio.py", line 205, in start File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 608, in run_forever File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\base_events.py", line 1936, in _run_once File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.2544.0_x64__qbz5n2kfra8p0\Lib\asyncio\events.py", line 84, in _run File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 545, in dispatch_queue File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 534, in process_one File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 437, in dispatch_shell File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\ipkernel.py", line 362, in execute_request File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\kernelbase.py", line 778, in execute_request File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\ipkernel.py", line 449, in do_execute File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\ipykernel\zmqshell.py", line 549, in run_cell File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3077, in run_cell File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3132, in _run_cell File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\async_helpers.py", line 128, in _pseudo_sync_runner File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3336, in run_cell_async File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3519, in run_ast_nodes File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\IPython\core\interactiveshell.py", line 3579, in run_code File "C:\Users\deanc\AppData\Local\Temp\ipykernel_21724\1451080196.py", line 1, in <module> File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 562, in predict File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 259, in one_step_on_data_distributed File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 249, in one_step_on_data File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\trainer.py", line 104, in predict_step File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\layer.py", line 908, in __call__ File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\operation.py", line 46, in __call__ File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 156, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\models\sequential.py", line 213, in call File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\models\functional.py", line 182, in call File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\function.py", line 171, in _run_through_graph File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\models\functional.py", line 637, in call File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\layer.py", line 908, in __call__ File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 117, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\operation.py", line 46, in __call__ File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\utils\traceback_utils.py", line 156, in error_handler File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py", line 146, in call File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\ops\numpy.py", line 168, in add File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\sparse.py", line 493, in sparse_wrapper File "C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\backend\tensorflow\numpy.py", line 60, in add Matrix size-incompatible: In[0]: [32,15], In[1]: [210,128] [[{{node sequential_151_1/dense_935_1/BiasAdd}}]] [Op:__inference_one_step_on_data_distributed_9572532]
Don't understand why it's not working anymore, going to skip this code because it's not worth my fucking time no more.
In [ ]:
ywaist_pred = modelwaist_49.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
In [ ]:
ywaist_pred = modelwaist_50.predict(Xwaist_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_test, label='True waist')
ax.legend();
6/6 ━━━━━━━━━━━━━━━━━━━━ 0s 5ms/step
Despite the increase of 14 data points at the end, the missing last bit is definitely influential to the model's risk! Having the sequential/window period data gives the model a lot of information to learn patterns. We'll revisit the show_score function to work with waist_sequence data.
In [ ]:
show_scores(modelwaist_49, ywaist_sequence_test, expandX_test, 'Waist Evaluation'), show_scores(modelwaist_50, ywaist_sequence_test, expandX_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 39ms/step 5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 22ms/step
Out[ ]:
(('Waist Evaluation',
{'Test MAPE (%)': np.float64(134.8436578920531),
'Test MSE': 8499.11331141435,
'Test R^2': -0.2845085176455995}),
('Waist Evaluation',
{'Test MAPE (%)': np.float64(163.04135949807323),
'Test MSE': 10209.751725000777,
'Test R^2': -0.5430448534198851}))
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
ywaist_pred = modelwaist_50.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
The MAPE of the model performs worse than models that don't predict, but want the variations and risk of it. So we'll keep MODEL_49
MODEL_51 - Patience 75¶
Gonna increase patience, but keep delta the same.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_51 = keras.Sequential([
layers.Dense(128, activation=None, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=75, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_51.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=150),
metrics=['mae'])
hist_51 = modelwaist_51.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [ ]:
plt.plot(hist_51.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
In [ ]:
show_scores(modelwaist_51, ywaist_sequence_test, expandX_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
Out[ ]:
('Waist Evaluation',
{'Test MAPE (%)': np.float64(126.9631042586312),
'Test MSE': 8542.257855992337,
'Test R^2': -0.29102914314731354})
MODEL_52 - Extra neuron layer¶
Tryiing another extra layer with more neurons, to see if it further helps with accuracy.
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_52 = keras.Sequential([
layers.Dense(256, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(128),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=75, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_52.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=150),
metrics=['mae'])
hist_52 = modelwaist_52.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
In [ ]:
plt.plot(hist_52.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[ ]:
Text(0.5, 1.0, 'Loss over Epoch')
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
show_scores(modelwaist_52, ywaist_sequence_test, expandX_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 35ms/step
Out[ ]:
('Waist Evaluation',
{'Test MAPE (%)': np.float64(118.76945109124502),
'Test MSE': 7990.06780642193,
'Test R^2': -0.20757422308174012})
In [ ]:
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.optimizers import Adam
from tensorflow.keras.losses import Huber
# set seed
tf.random.set_seed(35)
expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
# create model including relu and linear
modelwaist_52 = keras.Sequential([
layers.Dense(256, input_shape=(expandX_train.shape[1],)),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(128),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(64),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dropout(0.2),
layers.Dense(32),
layers.BatchNormalization(),
layers.Activation('tanh'),
layers.Dense(1, activation='linear')
])
reduce_lr = ReduceLROnPlateau(monitor='loss', # Track loss (can also use 'val_loss')
factor=0.5, # Reduce LR by half when triggered
patience=75, # Wait 10 epochs before reducing
min_lr=1e-6) # Do not reduce below this
modelwaist_52.compile(optimizer=Adam(learning_rate=0.001),
loss=Huber(delta=150),
metrics=['mae'])
hist_52 = modelwaist_52.fit(expandX_train, ywaist_sequence_train, epochs=1000, verbose=0, callbacks=[reduce_lr])
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\core\dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(activity_regularizer=activity_regularizer, **kwargs)
To be honest, from this point onwards, there's just very little incremental changes, that copy pasting and changing one singular line of code is taking up way too much space on this work book. So I'll be compiling all the changes I've made in order, why I made those changes, and will note any significant details.
MODEL_55
# ADDED code layers.Dense(256, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)), # Add L2 regularization
Chatgpt suggested L2 regularization makes the model more flexible, and does show in the loss history
MODEL_56
# ADDED code layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)), layers.BatchNormalization(), layers.Activation('tanh'), layers.Dropout(0.1),
The 256 neuron layer did slightly better with the added complexity, so made another layer on top of that to 512. It created spikes of loss increases, but comes back down, though still improves loss overall
MODEL_57
# TRIED code optimizer=Adam(learning_rate=0.01)
Increase learning rate of
Adamto allow model to add more variance. Model stagnated aroundloss 500between epoch 200-500 and didn't have extent of improvement as 56MODEL_58
# ADDED code # change all dropout to 0.1 layers.Dropout(0.1),
All remaining
Dropoutset from 0.2, to 0.1. Makes model more prone to overfitting, and increases it's risk taking. Overfitting is taking effect slightly with decrease in lossMODEL_59
# TRIED code def cyclic_lr(epoch, lr): base_lr = 1e-4 max_lr = 1e-2 step_size = 2000 cycle = np.floor(1 + epoch / (2 * step_size)) x = np.abs(epoch / step_size - 2 * cycle + 1) lr = base_lr + (max_lr - base_lr) * np.maximum(0, (1 - x)) return lr # Create the callback lr_scheduler = LearningRateScheduler(cyclic_lr, verbose=1)
Try Cyclic Learning Rate (CLR). Rather than having lr going down, this one goes up and down in a specific range of values for lr. Something like sine waves. Heavy increase spike of loss and lack of improvement
MODEL_60
# ADDED code modelwaist_53.fit(expandX_train, ywaist_sequence_train, batch_size=64, epochs=1000, verbose=0, callbacks=[reduce_lr])
Batch size is used to define how many training examples are used to compute the gradient/predictions. There's some slight improvement
MODEL_61
# TRIED code patience=20
ChatGPT suggested that patience 75 is too high, and try something between 15 to 30. So Tried 20. Sopped the random spike of loss increase, but loss is not as low as before
MODEL_61
# TRIED code scaler = StandardScaler() expandX_train = scaler.fit_transform(expandX_train)
Feature engineer by normalizing training/testing data. Had similar loss, but had trouble with predicting data on test data, with
MAPEreaching +200MODEL_62
# ADDED code layers.LeakyReLU(alpha=0.1),
Trying a different activation function with
LeakyReLU. Had success with it along withTanh. Was really good at decreasing Loss, but seem to overfit too much asMAPEwas at 200 for test dataMODEL_63
# ADDED code layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.001)), layers.Dropout(0.3),
Due to overfitting, I wanna decrease it a bit. ChatGPT suggests reducing
l2 regularizer, and increaseDropoutto improve regularization. Still same issue, where improves really fast, and then stops improvement quite drasticallyMODEL_64
# TRIED code early_stopping = EarlyStopping(monitor='val_loss', patience=100, restore_best_weights=True)
Adding early stopping to prevent the model from overfitting, after it stops improving. Didn't really activate so didn't do its job
MODEL_65
# ADDED code layers.ActivityRegularization(l2=0.01),
Another activation function, which prevents overfitting, discouraging overly large activations. Similar output, and the graph visual looks eerily similar to the true data, except being offsetted by 1 or 2 days
MODEL_66
# REMOVED code batch_size=64
Thought this code might cause some sort of data leakage, and wanna see results if removed. No change
Data leakage?¶
I'm wondering there's some sort of data leakage going on, as it looks like the predicted data is offset from the true data by a few points. So I'll be running a few code to have a look. Firstly, I'll check if there's lag on prediction.
Code has returned that there is a lag of -2, where it's predicted about 2 indexes on average after the true data. My next step, is looking at data importance. So there are these two specific features that cause very positive and very negative correlation to each other, but the data it's trained on expandX_train has 210 different features, likely because of the sequence data of a moving 14 day average or some shit. So I have no idea how to really interpret the graph, other than some feature causing the model some heavy reliance upon it.
I think I'll once again remove sequence data and see how the model plays out.
In [ ]:
# Check prediction lag
import numpy as np
import matplotlib.pyplot as plt
# Flatten the data before cross-correlation
true_data = ywaist_sequence_test.flatten()
predicted_data = ywaist_pred.flatten()
# Compute cross-correlation
def cross_correlation(true_data, predicted_data):
correlation = np.correlate(true_data, predicted_data, mode='full')
lag = np.argmax(correlation) - len(true_data) + 1
return lag
# Compute cross-correlation
lag = cross_correlation(true_data, predicted_data)
print(f"Optimal lag is: {lag}")
Optimal lag is: -2
In [ ]:
import lime
from lime import lime_tabular
# Initialize LIME Explainer
explainer = lime_tabular.LimeTabularExplainer(expandX_train, training_labels=ywaist_sequence_train, mode='regression')
# Choose a data point to explain
i = 42 # index of the data point to explain
exp = explainer.explain_instance(expandX_test[i], modelwaist_53.predict)
# Visualize the explanation
exp.show_in_notebook()
157/157 ━━━━━━━━━━━━━━━━━━━━ 0s 2ms/step
In [ ]:
expandX_test = expandX_test.reshape(expandX_test.shape[0], -1)
ywaist_pred = modelwaist_53.predict(expandX_test)
fig, ax = plt.subplots(figsize=(20,10))
ax.set(title='Waist measurement difference prediction', xlabel='Date')
ax.plot(ywaist_pred, label='Waist prediction')
ax.plot(ywaist_sequence_test, label='True waist')
ax.legend();
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
This is the kind of result's I've been seeing so far with experimenting
MODEL_67
# REMOVED code expandX_train = expandX_train.reshape(expandX_train.shape[0], -1)
- See if this sequence data of expanding X_train causes the data leakage. And yup it seems to be the case, as loss overtime was decrease at a more constant pace, rather than a sharp drop off close to 0
MODEL_68
# ADDED code expandX_train = expandX_train.reshape(expandX_train.shape[0], -1) # CHANGED code to def create_sequences(X, y, window_size): Xs, ys = [], [] for i in range(len(X) - window_size): Xs.append(X[i:i+window_size]) ys.append(y[i]) # Fixed code return np.array(Xs), np.array(ys)
- Had ChatGPT, help form me a new sequence function that will result in no lag and no data leakage
ys.append(y[i:i+window_size]) # OLD ys.append(y[i]) # NEW
- With the y sequence
i+window_size, this basically stated the y value on the index, plus the window_size. Meaning we get a re-formatted y data, wherethe model is predicting the future value, instead of the current value. This makes it easy for the model to predict, as the data is leaked from X, because it has access to future data that it wasn't supposed to have - After the change, the model now makes reasonable predictions!
MODEL_69
# CHANGED code layers.Dense(64), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.3), layers.ActivityRegularization(l2=0.01), layers.Dense(32), layers.LeakyReLU(alpha=0.1), layers.ActivityRegularization(l2=0.01),
- Removed Batch Normalization layers from the last two Dense layer. ChatGPT says it might cause distortions on final predictions. Doesn't seem to affect much on the loss or prediction
MODEL_70
# CHANGED code layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.001)), layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.3), layers.Dense(256), layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.3), layers.Dense(128), layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.3),
- Removed
ActivityRegularizationlayers where there are batch normalzation layers. It's likely not nessecary as dropout already does regularization. Prediction does seem to slightly be more riskier, and shows loss improvement
- Removed
MODEL_71
# CHANGED code layers.Dense(512, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)), layers.Dropout(0.1),
- Reduce dropout, and increase regularizer l2. These were changed because of overfitting data, but because of
sequence functiondata leak and fixing it up, it was underfitting and not making risky predictions. It's now overfitting heavily
- Reduce dropout, and increase regularizer l2. These were changed because of overfitting data, but because of
MODEL_72
layers.Dropout(0.2)
- Increase dropout to reduce overfitting, and shows slight improvement
MODEL_73
# CHANGED code loss=Huber(delta=50)
- Reduced Huber, model prediction is all over the place. It's being risky, but doeszn't seem to predict the data well
MODEL_74
# ADDED code hist_53 = modelwaist_53.fit(expandX_train, ywaist_sequence_train, validation_split=0.2, epochs=1000, verbose=0, callbacks=[reduce_lr])
- Added validation split, set to 0.2. 80% Training, and 20% Validation on the training data provided. Seem to improve model training, but flattens very quickly. Looks like it's overfitting
MODEL_75
# CHANGED code epochs=500
- The model has been getting really good at overfitting, and epoch flattens out considerably. Prediction is slightly better
MODEL_76
# CHANGED code patience=50
- Wanted to see what changes, if reduced from 75 to 50. During test data graph, it seemed like the prediction values are pulled down by a considerable margin from the true values. Say if prediction is 100, it would be -200 which is strange
MODEL_77
# ADDED code scaler = StandardScaler() expandX_train = scaler.fit_transform(expandX_train) ywaist_sequence_train = scaler.fit_transform(ywaist_sequence_train.reshape(-1, 1)) # reshape from 1D to 2D, as scaler only accepts 2D
- ChatGPT suggested to try scaling values as it usually helps with prediction. The model did pretty well with predictions around the middle, but struggle with the outliers.
MAPEwhich is what I was using to judge prediction accuracy for each model, is now messed up as its588instead of the rough130-200the model was getting. Likely due to the scaler changing y axis values
- ChatGPT suggested to try scaling values as it usually helps with prediction. The model did pretty well with predictions around the middle, but struggle with the outliers.
MODEL_78
# CHANGED code layers.Activation('tanh'),
- Still remembered tanh do well with values around 0, and the scaler should be perfect for tanh. It learned training data with zero effort, immediately flattening out the epoch, but did poorly at predictions, likely due to overfitting
MODEL_79
# CHANGED code epochs=100
- Due to how quick the model is learning patterns from the training data, I've further reduced epochs to 100 to prevent overfitting. Even though loss is close to 0 and has loss all meaning of whether the loss over epoch graph is indicative to the model's performance, the prediction is very poor, showing very little risk taking from the model
MODEL_80
# CHANGED code layers.LeakyReLU(alpha=0.1) epochs=1000
- I returned to 1000 epochs, realizing despite the epoch loss flattens, it doesn't mean a consistent result for the model prediction. Teetering back to LeakyReLU as well. With 1000 epochs, it does show the model taking the proper risks
MODEL_81
# CHANGED code layers.Activation('tanh'),
- Again coming back to tanh, just to see it's prediction with 1000 epochs. MAPE has improved, but is not risking prediction values. We'll stick with LeakyReLU from now on
MODEL_81
# ADDED/CHANGED code layers.Dense(1024, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)), layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.2), layers.Dense(512),
- Create an even denser model to see if it can improve performance. Doesn't seem to make much improvement
MODEL_82
# ADDED code lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay( initial_learning_rate=0.001, decay_steps=1000, decay_rate=0.9 ) modelwaist_53.compile(optimizer = Adam(learning_rate=lr_schedule),...)
- Applied exponential learning rate decay. It's used to help the model learn quickly at first with a higher lr, and has better stability as lowering lr precents overshooting of optimal solution
MODEL_83
# USED code import numpy as np errors = np.abs(expandX_test - ywaist_sequence_test) # Compute absolute errors print(np.percentile(errors, [25, 50, 75, 90])) # Look at error percentiles # RESULT [0.33977055 0.68632555 1.34403686 1.84173002] loss=Huber(delta=1.8),
- What the code above does, is telling me the percentile of where the error occurs. 90, means 90% of my errors are at 1.8 unit or below. 50, is 50% of error at 0.6 unit or below. This helps indicate the delta I should use for huber. As my data has changed from x100 of original dataset, to now scaled, the delta should change accordingly to match the value distribution of my new data. So I would set my Delta to 1.8
MODEL_84
# CHANGED code layers.Dense(256, input_shape=(expandX_train.shape[1],), kernel_regularizer=regularizers.l2(0.01)), # REMOVED code layers.Dense(512), layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.2), layers.Dense(256), layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.2),
- I've deleted two layer bunches, as the model seems to overfit a lot on the training data, and not exactly make good predictions
MODEL_85
# CHANGED code decay_steps=500, # REMOVED code reduce_lr = ReduceLROnPlateau(monitor='loss', factor=0.5, patience=100, min_lr=1e-6)
- Exponential decay might be too slow, so we'll have it reduce the learning rate at a faster speed. Also encountered an error where both exponential decay and reduce_lr conflict each other, with how they change lr. So removed reduce_lr, to see how exponential decay works
MODEL_86
# ADDED code layers.GaussianNoise(0.05, input_shape=(expandX_train.shape[1],)),
- Adding a layer of gaussian noise, which can help prevent model from over relying on exact training values. Should help it generalize better
MODEL_87
# CHANGED code loss=tf.keras.losses.LogCosh()
- Replaced Huber with LogCosh. This loss function allows the model to take bigger prediction steps without being heavily punished, e.g. outlier points
MODEL_88
# REMOVED code layers.ActivityRegularization(l2=0.01), # CHANGED code layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)), layers.Dense(256), layers.Dense(128), layers.Dense(64),
- Activity Regularization is removed, as it performs a similar function to batch normalization, and having a lot of these kinds of layers especially near the end, gets redundant. I've also moved Density of neurons up by one exponent, and shrunk regularizer.l2 from 0.01 to 0.001, as it could be too strong and only predicting conservatively
MODEL_89
# CHANGED code layers.Dropout(0.6), decay_steps=200,
Seems like I'm reaching a wall in terms of prediction, as it seems to not really have much affect on prediction, likely due to how there's many more factors that also play on weight change from a day by day basis, such as water weight, sodium intake (which affects amount of water weight), timing of food intake, exercise, etc.
I'm just curious what a very high dropout, and very low decay step for lr_schedule is going to do
MODEL_90
# CHANGED code layers.GaussianNoise(0.1, input_shape=(expandX_train.shape[1],)), layers.Dropout(0.1),
- Put dropout down to 0.1, as it stops learning to a point, and seems to make very cautious predictions. Gaussion is increased from 0.05 to 0.1
MODEL_91
# ADDED code from tensorflow.keras import backend as K def weighted_log_cosh(y_true, y_pred): error = y_pred - y_true # calculate difference/error of pred and truth weight = K.exp(K.abs(error)) # apply exponents to absolute error, creating more emphasis on large errors return K.mean(weight * K.log(tf.cosh(error + 1e-12))) # tf.cosh(error) > cosine of the error, used for smoothing approximation of error, K.log() > smoothe the penalty,1e-12 > very small value added to avoid calculations at 0
- This block of code, helps penalize under predictions, which should get it to predict more riskier, and it somewhat did what was advertised
MODEL_92
# CHANGED code lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts( initial_learning_rate=0.001, first_decay_steps=200, # Will start decaying learning rate, after 200 epochs t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay) m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr alpha=1e-6 # Minimum LR )
- lr_scheduler is changed to CosineDecayRestarts. Code forces model to restart learning at periodic intervals, escaping local minima, and encourages bolder updates throughout training
MODEL_93
# CHANGED code layers.Dense(1, activation='tanh')
- I've used tanh on LeakyReLU layers, but never on the last linear layer. So let's give that a go. Not the best results, as prediction laid between 1 and -1, but standard deviation values have reach numbers past 3 and -3
MODEL_94
# CHANGED code layers.Dense(1)
- Changed it to not include tanh or linear. Model has been encouraged to make riskier predictions slightly
MODEL_95
# CHANGED code weight = K.log(1 + K.exp(K.abs(error)))
- ChatGPT suggested change to
weight_log_coshfunction, as it was penalizing extreme predictions too harshly, and this should help, by decreasing/softening the weight of extreme values. It is taking more risks now, and the loss error graph stopped showing how it's reaching 0 loss immediately, but gradually instead
- ChatGPT suggested change to
MODEL_96
# REMOVED code layers.Dropout(0.1)
- Curious as to see the results, predictions became a bit reserved
MODEL_97
# CHANGED code weight = K.log(5 + K.exp(K.abs(error)))
- k.log(was 1), now it is 5. This is done as previously stated, we wanted to soften the punishment on extreme values. And having that 1, basically elevated all smaller value with a +5, making the errors of the extreme value even smaller. Model is showing to take risks more
MODEL_98
# ADDED code def quantile_loss(q): def loss(y_true, y_pred): e = y_true - y_pred # calculate error difference between pred and truth return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value return loss loss=quantile_loss(0.9), # REMOVED code def weighted_log_cosh(y_true, y_pred): error = y_pred - y_true # calculate difference/error of pred and truth weight = K.log(5 + K.exp(K.abs(error))) # apply exponents to absolute error, creating more emphasis on large errors return K.mean(weight * K.log(tf.cosh(error + 1e-12))) # tf.cosh(error) > cosine of the error, used for smoothing approximation of error, K.log() > smoothe the penalty,1e-12 > very small value added to avoid calculations at
- gpt suggested to try another loss function called
quantile_loss. It has a value calledq, where you can change to penalize either underpredicton (higher value), or overprediction (lower value) more. So i chose 0.9 as the starting point. Some of the riskiest prediction I've seen!0
- gpt suggested to try another loss function called
MODEL_99
# CHANGED code loss=quantile_loss(0.7),
- First time saying that the model is playing a bit risky, and need to tone down a bit.
Final Verdict on Model¶
After 99 iterations of the model, I think I've found what is the most ideal model. It's not the best prediction results, likely due to how daily changes to waist and weight are affected by more than just current weight/waist, macros and calories.
Here's is the fixed sequence function, where the y data is changed to not have any data leak (something I've mentioned a while ago).
In [188]:
# Adjust by removing the lag by setting ys.append(y[i]) instead of y[i+window_size]
def create_sequences(X, y, window_size):
Xs, ys = [], []
for i in range(len(X) - window_size):
Xs.append(X[i:i+window_size])
ys.append(y.iloc[i]) # Fix the lag by predicting current time step. It predicts the first value of the window_size, rather than a future value, since we added the window_size to the index.
return np.array(Xs), np.array(ys)
window_size=7 # size of how far our data will catch trends for (in days)
Xwaist_sequence_train, ywaist_sequence_train = create_sequences(Xwaist_train, ywaist_train, window_size)
Xwaist_sequence_test, ywaist_sequence_test = create_sequences(Xwaist_test, ywaist_test, window_size)
Final Model¶
In [190]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
from tensorflow.keras import regularizers
from tensorflow.keras.optimizers import Adam
# Set seed for reproducibility
tf.random.set_seed(35)
# Ensure input shape is correct
Xwaist_sequence_train = Xwaist_sequence_train.reshape(Xwaist_sequence_train.shape[0], -1)
scaler_X_waist = StandardScaler()
normX_waist_train = scaler_X_waist.fit_transform(Xwaist_sequence_train)
scaler_y_waist = StandardScaler()
normY_waist_train = scaler_y_waist.fit_transform(ywaist_sequence_train.reshape(-1, 1))
# Define model
modelwaist = keras.Sequential([
layers.GaussianNoise(0.1, input_shape=(normX_waist_train.shape[1],)),
layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1)
])
lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.001,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6 # Minimum LR
)
def quantile_loss(q):
def loss(y_true, y_pred):
e = y_true - y_pred # calculate error difference between pred and truth
return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss
modelwaist.compile(optimizer = Adam(learning_rate=lr_schedule),
loss=quantile_loss(0.7),
metrics=['mae'])
hist = modelwaist.fit(normX_waist_train, normY_waist_train,
validation_split=0.2,
epochs=1000, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
The loss history of training the model, after 1000 epochs¶
In [191]:
plt.plot(hist.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[191]:
Text(0.5, 1.0, 'Loss over Epoch')
Showing scores of model¶
In [192]:
Xwaist_sequence_test = Xwaist_sequence_test.reshape(Xwaist_sequence_test.shape[0], -1)
normX_waist_test = scaler_X_waist.transform(Xwaist_sequence_test)
normY_waist_test = scaler_y_waist.transform(ywaist_sequence_test.reshape(-1, 1))
show_scores(modelwaist, normY_waist_test, normX_waist_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 23ms/step
Out[192]:
('Waist Evaluation',
{'Test MAPE (%)': np.float64(403.76744447706216),
'Test MSE': 1.0789613768209771,
'Test R^2': -0.14378946955523975})
Monte Carlo predictions¶
What this function does, is it simulates a number of predictions on the test results. It returns the mean prediction, along with the range of 2 standard deviation.
In [193]:
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_waist, uncertainty = monte_carlo_predictions(modelwaist, normX_waist_test, n_simulation=100)
Model Prediction vs Truth Graph¶
I'll create a graph, to display the true values of waist difference, along withmonte carly predictions (mean prediction + 2 standard deviation)
In [194]:
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(range(len(mean_pred_waist)), mean_pred_waist, label="Waist Prediction", color='blue')
plt.plot(normY_waist_test, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist)), mean_pred_waist - 2 * uncertainty, mean_pred_waist + 2 * uncertainty,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
Weight Prediction¶
Now that we've got the prediction of Waist, I tried it on the Weight data.
I did some fixing up with variable names and doing some cleaning/reorganizing of code. However it's underpredicting compared to the previous model with the unorganized variable names. So will list the changes like the waist model to better model:
MODEL_1
# CHANGED code epochs=100
- Change epoch=1000 to 100. This is so testing and visualizing prediction trends be very quick. Unadvertedly made the model take more risks, so will be playing with epochs
MODEL_2
# CHANGED code epochs=200
- 100 to 200 epochs to see if risk is increasing
MODE_3
# CHANGED code loss=quantile_loss(0.9), epochs=300
- 200 to 300 epochs, and increase quantile_loss to hopefully promote more risk taking, not much change
MODEL_4
# CHANGED code loss=quantile_loss(0.1),
- Minimize quantile_loss to see results. Underpredicted
MODEL_5
# CHANGED code initial_learning_rate=0.005, loss=quantile_loss(1),
- *Higher learning rate to help with larger updates on weights over tiny ones, and set quatile loss at highest, to 1. Prediction is somehow made around value
10over0
- *Higher learning rate to help with larger updates on weights over tiny ones, and set quatile loss at highest, to 1. Prediction is somehow made around value
MODEL_6
# CHANGED code loss=quantile_loss(0.9),
- Reduce quantile_loss back to 0.9, model predicts as it should
MODEL_7
# CHANGED code epochs=1000
- Increase epoch back to 1000, seems like model's trajectory continues treading down. Model takes more risks but seems a bit erratic in prediction
MODEL_8
# CHANGED code layers.Dropout(0.3),
- Increase dropout to see if it helps with erratic prediction and generalization, model is not predicting negative values
MODEL_9
# CHANGED code initial_learning_rate=0.001, layers.Dropout(0.2),
- *Reduced dropout, and reused the same initial learning_rate since it may have cause such prediction troubles. Now I realize if initial lr is high, it artificially pulls up the prediction
MODEL_10
# CHANGED code initial_learning_rate=0.0005,
- *Reduced initial lr to prevent artificial pulling up of prediction values
MODEL_11
# ADDED code layers.Dense(1024, kernel_regularizer=regularizers.l2(0.001)), # More neurons layers.BatchNormalization(), layers.LeakyReLU(alpha=0.1), layers.Dropout(0.2),
- Creating another dense layer batch, with 1024 neurons. Moving regularizers l2 to the
1024layer, performance was mediocre
- Creating another dense layer batch, with 1024 neurons. Moving regularizers l2 to the
MODEL_12
# CHANGED code layers.Dense(512, kernel_regularizer=None),
- kernel_regularizer set to none, chatgpt suggests the l2 regularizer may constrain model's ability to aggressively explore patterns, aka take more risks. Changes not significant
MODEL_13
# CHANGED code layers.Dropout(0.1),
- setting dropout back to 0.1, again lack of change
In [122]:
Xweight_sequence_train, yweight_sequence_train = create_sequences(Xweight_train, yweight_train, window_size)
Xweight_sequence_test, yweight_sequence_test = create_sequences(Xweight_test, yweight_test, window_size)
In [131]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
# Set seed for reproducibility
tf.random.set_seed(35)
# Ensure input shape is correct
Xweight_sequence_train = Xweight_sequence_train.reshape(Xweight_sequence_train.shape[0], -1)
scaler_X_weight = StandardScaler()
normX_weight_train = scaler_X_weight.fit_transform(Xweight_sequence_train)
scaler_y_weight = StandardScaler()
normY_weight_train = scaler_y_weight.fit_transform(yweight_sequence_train.reshape(-1, 1))
# Define model
modelweight = keras.Sequential([
layers.GaussianNoise(0.1, input_shape=(normX_weight_train.shape[1],)),
layers.Dense(512, kernel_regularizer=None),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1)
])
lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.0005,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6 # Minimum LR
)
def quantile_loss(q):
def loss(y_true, y_pred):
e = y_true - y_pred # calculate error difference between pred and truth
return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss
modelweight.compile(optimizer = Adam(learning_rate=lr_schedule),
loss=quantile_loss(0.9),
metrics=['mae'])
hist = modelweight.fit(normX_weight_train, normY_weight_train,
validation_split=0.2,
epochs=1000, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [132]:
plt.plot(hist.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[132]:
Text(0.5, 1.0, 'Loss over Epoch')
In [165]:
Xweight_sequence_test = Xweight_sequence_test.reshape(Xweight_sequence_test.shape[0], -1)
normX_weight_test = scaler_X_weight.transform(Xweight_sequence_test)
normY_weight_test = scaler_y_weight.transform(yweight_sequence_test.reshape(-1, 1))
show_scores(modelweight, normY_weight_test, normX_weight_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 13ms/step
Out[165]:
('Waist Evaluation',
{'Test MAPE (%)': np.float64(370.2774930790802),
'Test MSE': 2.08010199127118,
'Test R^2': -0.13372795895491119})
In [166]:
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_weight, uncertainty_weight = monte_carlo_predictions(modelweight, normX_weight_test, n_simulation=100)
In [167]:
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(range(len(mean_pred_weight)), mean_pred_weight, label="Weight Prediction", color='blue')
plt.plot(normY_weight_test, label='True weight', color='red')
plt.fill_between(range(len(mean_pred_weight)), mean_pred_weight - 2 * uncertainty_weight, mean_pred_weight + 2 * uncertainty_weight,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
Results are okay, I mean it seems to underfit a bit especially near the latter portion of the data, but I say it works fine for now.
Model performance on training data???¶
I'm bummed out how hard it is to get the model to take risks in its predictions, and don't understand how many tries in perfecting hyperparameters, layers, and functions, it doesn't make too much of a dent in getting the model to risk take more.
So now I'm curious, what does the model look like if I use the model to predict the training data instead, and compare it to the actual train labels.
In [136]:
# mean model prediction of waist train
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_waist_train, uncertainty = monte_carlo_predictions(modelwaist, normX_waist_train, n_simulation=10)
In [137]:
# graph plot of waist training data
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(normY_waist_train, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist_train)), mean_pred_waist_train - 2 * uncertainty, mean_pred_waist_train + 2 * uncertainty,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.plot(range(len(mean_pred_waist_train)), mean_pred_waist_train, label="Waist Prediction", color='blue')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
Oh look! There's definitely some anomaly as the model continues down the data, especially from 1200 index and beyond, where the model takes a much more cautious prediction.
Now lets see the weight trianing data
In [168]:
# mean prediction weight train
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_weight_train, uncertainty_weight = monte_carlo_predictions(modelweight, normX_weight_train, n_simulation=10)
In [169]:
# weight train graph display
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(normY_weight_train, label='True weight', color='red')
plt.fill_between(range(len(mean_pred_weight_train)), mean_pred_weight_train - 2 * uncertainty_weight, mean_pred_weight_train + 2 * uncertainty,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.plot(range(len(mean_pred_weight_train)), mean_pred_weight_train, label="Weight Prediction", color='blue')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
Jesus, it's even more prominent in the weight data! The underprediction range of the data, is similar to the one in the waist data, starting around index 1200.
Let's see the actual waist values, aka the real waist size day by day rather than the waist diffference, when compared to the prediction values of the model. I'll visualize the waist size, from index 800 to 1500. Even though the anomaly starts around index 1200, I also want to see a number of previous indexes before the anomaly to see if the waist data provides any visual cues towards the anomaly.
In [140]:
# Plotting the waist values between indexes 800-1500
import matplotlib.pyplot as plt
plt.plot(clean_df['Waist (cm)'][800:1500])
plt.title('Raw Waist Values - Index 800-1500')
plt.xlabel('Index')
plt.ylabel('Waist')
plt.legend()
plt.grid(True)
plt.show();
C:\Users\deanc\AppData\Local\Temp\ipykernel_53232\3308172792.py:8: UserWarning: No artists with labels found to put in legend. Note that artists whose label start with an underscore are ignored when legend() is called with no argument. plt.legend()
We can see where the model underpredicts, is around the period of time where the waist data transitions from stablility, to a persistent increase. Let's see how the other data from our X data compare between index 800 to 1500 as well.
Namely we'll look at weight and calorie consumption, then display weight prediction graph as a comparison as well. (Not macros, as from previous visualizations, they don't have as heavy affect on waist/weight change compared to caloric intake)
In [170]:
# display weight data, caloric consumption, and pred/true values of weight change
fig, ax = plt.subplots(nrows=3, figsize=(20,15), sharex=True)
ax[0].set(title='Weight value', xlabel='Index')
ax[0].plot(clean_df['Weight (kg)'][800:1500], label='Weight data')
ax[1].set(title='Caloric consumption', xlabel='Index')
ax[1].plot(clean_df['Daily intake (kcal)'][800:1500], label='Daily kcal intake')
ax[2].set(title='Model prediction vs weight fluctuation', xlabel='Index')
ax[2].plot(range(800,1450), normY_weight_train[800:1450], label='True values') # (range) to only select data of variable from 800-1500. If i did what i did with ax[1]/[0], it'll cut the front out, and bring everything to index 0
ax[2].plot(range(800,1450), mean_pred_weight_train[800:1450], label='Model prediction')
ax[2].legend();
Yup, it does seem like the change in weight/waist increase and the increase in caloric consumption around index 1200, seem to cause the model to underpredict from there on, and I can see why.
Change in lifestyle:¶
For the most part from index 0 to 1200, my lifestyle was usually very sedentary, and was very home bound. What changed from 1200 was I started to go to the gym 5-6 times week, being a lot more active in addition to eating a lot of food. Lot of these were very different from my old lifestyle, which is probably what my model picked up on during learning.
A lot of my data from the start made sense to it, due to the predictability of how much food I ate, the macronutrients, weight/waist variations etc. But then suddenly it hits a wall at around index 1200 when my caloric consumption was unusual from before, macros were unusual from before and for the most part, my weight and waist weren't following the patterns it was recognizing from the start.
What now?¶
Well, unfortunately there isn't much I can really do with it unless I collect enough data of my weight, waist etc. post lifestyle change, so the model has enough data points to learn the new patterns for it.
And once enough data is collected, I can create two separated data. One with my sedentary lifestyle, and the other with my active lifestyle, so the respective models for each data set can learn the patterns that relate to my current lifestyle in the data.
BUT!
Because I spent so much time in perfecting the model, we might as well give it a go, and only train the ml model with the active lifestyle portion of the data to see if the resulting prediction is better.
Retrain model on active lifestyle data portion:¶
In [201]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
# Set seed for reproducibility
tf.random.set_seed(35)
# Filter training data to start post index 1150, aka where active lifestyle starts
Xwaist_sequence_train_active = Xwaist_sequence_train[1150:]
ywaist_sequence_train_active = ywaist_sequence_train[1150:]
# Ensure input shape is correct
Xwaist_sequence_train_active = Xwaist_sequence_train_active.reshape(Xwaist_sequence_train_active.shape[0], -1)
scaler_X_waist = StandardScaler()
normX_waist_train_active = scaler_X_waist.fit_transform(Xwaist_sequence_train_active)
scaler_y_waist = StandardScaler()
normY_waist_train_active = scaler_y_waist.fit_transform(ywaist_sequence_train_active.reshape(-1, 1))
# Define model
modelwaist_active = keras.Sequential([
layers.GaussianNoise(0.1, input_shape=(normX_waist_train.shape[1],)),
layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1)
])
lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.001,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6 # Minimum LR
)
def quantile_loss(q):
def loss(y_true, y_pred):
e = y_true - y_pred # calculate error difference between pred and truth
return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss
modelwaist_active.compile(optimizer = Adam(learning_rate=lr_schedule),
loss=quantile_loss(0.7),
metrics=['mae'])
histwaist_active = modelwaist_active.fit(normX_waist_train_active, normY_waist_train_active,
validation_split=0.2,
epochs=1000, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [202]:
plt.plot(histwaist_active.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[202]:
Text(0.5, 1.0, 'Loss over Epoch')
In [203]:
Xwaist_sequence_test = Xwaist_sequence_test.reshape(Xwaist_sequence_test.shape[0], -1)
normX_waist_test = scaler_X_waist.transform(Xwaist_sequence_test)
normY_waist_test = scaler_y_waist.transform(ywaist_sequence_test.reshape(-1, 1))
show_scores(modelwaist_active, normY_waist_test, normX_waist_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 26ms/step
Out[203]:
('Waist Evaluation',
{'Test MAPE (%)': np.float64(235.0922096252269),
'Test MSE': 1.6186943022889198,
'Test R^2': -0.19643115107327835})
In [204]:
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_waist_active, uncertainty_active = monte_carlo_predictions(modelwaist_active, normX_waist_test, n_simulation=100)
In [205]:
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(range(len(mean_pred_waist_active)), mean_pred_waist_active, label="Waist Prediction", color='blue')
plt.plot(normY_waist_test, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist_active)), mean_pred_waist_active - 2 * uncertainty_active, mean_pred_waist_active + 2 * uncertainty_active,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
And this is the prediction on the entire dataset:
Doesn't seem to vary too much except the active model seem to underpredict a little more than the whole model. Despite that, it's still quite impressive how the active model can make such predictions with less than a year of data, when compared to whole model, with almost 5 times as much data as active.
Active model prediction on training data:¶
Since we've had a look at model's prediction on the whole training data, let's see the active model's prediction on it's training data.
In [335]:
# mean prediction of waist on the active data
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_waist_active_train, uncertainty_waist_train = monte_carlo_predictions(modelwaist_active, normX_waist_train_active, n_simulation=10)
In [336]:
mean_pred_waist_active_train.size
Out[336]:
306
In [339]:
# active train graph display
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(normY_waist_train_active, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_waist_active_train)), mean_pred_waist_active_train - 2 * uncertainty_waist_train, mean_pred_waist_active_train + 2 * uncertainty_waist_train,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.plot(range(len(mean_pred_waist_active_train)), mean_pred_waist_active_train, label="Waist Prediction", color='blue')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
plt.figure(figsize=(20,5))
plt.plot(clean_df['Waist (cm)'][1150:1150+306], label='True waist', color='red') # why +306 at last index? because that's how long the prediction data is, and would need to match it to get better visual between graph comparison
plt.xlabel("Sample Index")
plt.ylabel("Kilogram")
plt.title("Weight measurement overtime")
plt.legend()
plt.show()
plt.figure(figsize=(20,5))
plt.plot(clean_df['Daily intake (kcal)'][1150:1150+306], label='Calorie', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Kilocalories")
plt.title("Consumption of kcal overtime")
plt.legend()
plt.show()
Hmmmm... still the underprediction exists in the graph despite the data is inclusive of my active lifestyle :(
The last hypothesis for this could be because the model needs more training. Remember from the RNN data? There was still room for improvement in the prediction data, where there's notable spots that the model hasn't predicted, and first did parts that I assume are most notable to the model, then working it's way down to the more nuanced data points.
Regardless, I think it's a good idea to start wrapping it up, and see how the model's prediction compares once we convert it to day by day weight/waist measurements, to the actual measurements rather than the difference.
Unstandadized data¶
Since I've standardized the value of the train and test value, I'll need to reverse the process. This is so instead of having values of the weight and waist difference, I have a graph comparison on my weight, interchanged to my prediction, and comparing it to my true weight and waist, to see how it does overall with my actual weight and waist prediction.
The data was also multiplied by 100 times before, to make the model learn easier, rather than values sticking around 0.
Standardization errors¶
So I checked the results after reversing standardization, and realized an issue where the data that's reverse, did not match the actual true values at all. A little quick consultation on chatGPT said that, I should be creating separate scales for each y, x, train, test, weight, and waist data.
Normalization errors¶
I've gone back and decided to clean up the variable names to the normalization and scalers, as it used to look like this for my waist:
expandX_train = expandX_waist_train.reshape(expandX_train.shape[0], -1)
scaler_X = StandardScaler()
expandX_train = scaler.fit_transform(expandX_train)
scaler_y = StandardScaler()
y_sequence_train = scaler_y.fit_transform(y_sequence_train.reshape(-1, 1))
No labelling that it is waist data, and reusing same variables for scaling. Not the most professional/cleanest as there are reuse of same variables that might cause problems with trying to reverse standardization
Changed code:¶
expandX_train_waist = expandX_train_waist.reshape(expandX_train_waist.shape[0], -1)
scaler_X_waist_train = StandardScaler()
normX_waist_train = scaler_X_waist_train.fit_transform(expandX_train_waist)
scaler_y_waist_train = StandardScaler()
normY_waist_train = scaler_y_waist_train.fit_transform(ywaist_sequence_train.reshape(-1, 1))
Fit scalars on Test (Big nono)¶
I created separate scalars of standardization for every data, but I felt something was a bit off with doing what I was trying to do. Chat GPT has said that the scaling of test data, can cause data leakage as I was using fit_transform(). Rather, transform() should be used on the test data, and one scalar be used between the Train and Test sets of either X or y.
Redoing weight model¶
After fiddling with scalars and getting them right, the model is now underpredicting and being too cautious, so will need to do a bit of testing again.
I have redone the code of the final weight/waist model from the above. Now I'm ready to pursue with the rest of the code!
In [158]:
# reverse standardization of the data
y_pred_waist = scaler_y_waist.inverse_transform(mean_pred_waist.reshape(-1, 1)) # change the data into 2D array fr inverse transformation
y_true_waist = scaler_y_waist.inverse_transform(normY_waist_test) # data is already in 2D array
y_pred_weight = scaler_y_weight.inverse_transform(mean_pred_weight.reshape(-1, 1))
y_true_weight = scaler_y_weight.inverse_transform(normY_weight_test)
y values were multiplied by 100 before, so will need to divide back to original values
In [ ]:
y_pred_weight = y_pred_weight/100
y_true_weight = y_true_weight/100
y_pred_waist = y_pred_waist/100
y_true_waist = y_true_waist/100
In [195]:
# see if waist data matches
y_true_waist[:5],ywaist_sequence_test[:5]
Out[195]:
(array([[ 0.8],
[ 0.6],
[-0.2],
[-0.4],
[ 0.1]]),
array([ 80., 60., -20., -40., 10.]))
Prediction values vs. waist/weight measurements¶
Now I want to see how the model visually does compared to my actual waist/weight values, since it's hard to judge from the fluctuations of weight/waist differences.
Next step is to add the first weight/waist measurment value on to the first indexed prediction, and let it mathematecally operate on the oncoming prediction values.
In [160]:
start_waist = Xwaist_test['Waist (cm)'][:1]
start_weight = Xweight_test['Weight (kg)'][:1]
start_waist, start_weight
Out[160]:
(1463 73.2 Name: Waist (cm), dtype: float64, 1463 72.1 Name: Weight (kg), dtype: float64)
In [161]:
# convert prediction values to prediction of measurement instead
pred_waist_measure = [start_waist] # create a new list, and have starting value at the first waist measurement
for val in y_pred_waist: # create looping function to mathematically operate prediction values onto the first waist measurement
next_val = pred_waist_measure[-1] + val # get previous value and add/subtract the current value
pred_waist_measure.append(next_val)
pred_waist_measure = pred_waist_measure[1:] # remove the first value, to keep with same data shape of test
pred_waist_measure = np.array(pred_waist_measure)
pred_waist_measure = pred_waist_measure.flatten()
In [178]:
# convert prediction values to prediction of measurement instead
pred_weight_measure = [start_weight] # create a new list, and have starting value at the first weight measurement
for val in y_pred_weight: # create looping function to mathematically operate prediction values onto the first weight measurement
next_val = pred_weight_measure[-1] + val # get previous value and add/subtract the current value
pred_weight_measure.append(next_val)
pred_weight_measure = pred_weight_measure[1:] # remove the first value, to keep with same data shape of test
pred_weight_measure = np.array(pred_weight_measure)
pred_weight_measure = pred_weight_measure.flatten()
Prediction and true measurement comparison¶
In [196]:
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_waist_measure)), pred_waist_measure, label="Waist Prediction", color='blue')
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label='Waist Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Waist prediction converted to measurement vs waist measurment")
plt.legend()
plt.show()
plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_weight_measure)), pred_weight_measure, label="Weight Prediction", color='blue')
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label='Weight Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Weight prediction converted to measurement vs weight measurment")
plt.legend()
plt.show()
Observations:¶
Its clear that the full data model training does a steady continuous incline. The likely reasoning is due to the sedentary lifestyle contained a much heavier weight than the active lifestyle, due to how long I stayed sedentary during my data recording, but is definitely more evident in weight prediction.
So when it came to predicting my weight in the portion where I was in my active lifestyle, the majority of the pattern it has learnt was only the sedentary lifestyle. Naturally that would mean less caloric and macro consumption to keep a stable weight when sedentary, but is vice versa for an active lifestyle. Unfortunately the model did not understand the change in lifestyle, and treat my caloric and macro consumption as if I was still in my sedentary lifestyle.
But also another reasoning for the increase as well, could be seen in the way the model predicts training data. Despite matching well with the positive values, it just didn't do a good enough job with the negative values, which could likely have caused that imbalance.
Regarding that, it would be interesting to see the model make prediction on the training data with the true measurments, and see if it performs that over-estimation of weight/waist of the data.
In [197]:
# redo inverse standardization with the training data
y_pred_waist_train = scaler_y_waist.inverse_transform(mean_pred_waist_train.reshape(-1, 1)) # change the data into 2D array fr inverse transformation
y_true_waist_train = scaler_y_waist.inverse_transform(normY_waist_train) # data is already in 2D array
y_pred_weight_train = scaler_y_weight.inverse_transform(mean_pred_weight_train.reshape(-1, 1))
y_true_weight_train = scaler_y_weight.inverse_transform(normY_weight_train)
# divide 100 to get original data
y_pred_waist_train = y_pred_waist_train/100
y_true_waist_train = y_true_waist_train/100
y_pred_weight_train = y_pred_weight_train/100
y_true_weight_train = y_true_weight_train/100
start_waist_train = Xwaist_train['Waist (cm)'][:1]
start_weight_train = Xweight_train['Weight (kg)'][:1]
# convert prediction values to prediction of measurement instead
pred_waist_measure_train = [start_waist_train] # create a new list, and have starting value at the first waist measurement
for val in y_pred_waist_train: # create looping function to mathematically operate prediction values onto the first waist measurement
next_val = pred_waist_measure_train[-1] + val # get previous value and add/subtract the current value
pred_waist_measure_train.append(next_val)
pred_waist_measure_train = pred_waist_measure_train[1:] # remove the first value, to keep with same data shape of test
pred_waist_measure_train = np.array(pred_waist_measure_train)
pred_waist_measure_train = pred_waist_measure_train.flatten()
# convert prediction values to prediction of measurement instead
pred_weight_measure_train = [start_weight_train] # create a new list, and have starting value at the first waist measurement
for val in y_pred_weight_train: # create looping function to mathematically operate prediction values onto the first waist measurement
next_val = pred_weight_measure_train[-1] + val # get previous value and add/subtract the current value
pred_weight_measure_train.append(next_val)
pred_weight_measure_train = pred_weight_measure_train[1:] # remove the first value, to keep with same data shape of test
pred_weight_measure_train = np.array(pred_weight_measure_train)
pred_weight_measure_train = pred_weight_measure_train.flatten()
I also just realized, I could've just created a function to process the inverse standardization rather than copy pasting variable names. There's definitely a LOT of things I could've just moved into functions, and make this job a lot fuckin easier lmao.
In [198]:
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_waist_measure_train)), pred_waist_measure_train, label="Waist Prediction", color='blue')
plt.plot(range(len(Xwaist_train)), Xwaist_train['Waist (cm)'], label='Waist Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Waist prediction converted to measurement vs waist measurment")
plt.legend()
plt.show()
plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_weight_measure_train)), pred_weight_measure_train, label="Weight Prediction", color='blue')
plt.plot(range(len(Xweight_train)), Xweight_train['Weight (kg)'], label='Weight Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Weight prediction converted to measurement vs weight measurment")
plt.legend()
plt.show()
Some pretty unfortunate results sadly, as hypothesis is proven to be true.
Now I'd say it'd be worth it to also see the results for the active model as well.
In [206]:
pred_waist_active = scaler_y_waist.inverse_transform(mean_pred_waist_active.reshape(-1, 1))
pred_waist_active = pred_waist_active/100
# convert prediction values to prediction of measurement instead
pred_waist_measure_active = [start_waist] # create a new list, and have starting value at the first waist measurement
for val in pred_waist_active: # create looping function to mathematically operate prediction values onto the first waist measurement
next_val = pred_waist_measure_active[-1] + val # get previous value and add/subtract the current value
pred_waist_measure_active.append(next_val)
pred_waist_measure_active = pred_waist_measure_active[1:] # remove the first value, to keep with same data shape of test
pred_waist_measure_active = np.array(pred_waist_measure_active)
pred_waist_measure_active = pred_waist_measure_active.flatten()
In [207]:
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 5))
plt.plot(range(len(pred_waist_measure_active)), pred_waist_measure_active, label="Waist Prediction Active Model", color='blue')
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label='Waist Measurement', color='red')
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Waist prediction active model converted to measurement vs waist measurment")
plt.legend()
plt.show()
Observations:¶
Graph gives some slightly more favourable/reasonable prediction when it was trained on data that was mostly during my active lifestyle. But still seems to have that steady increase near the end portion of the data.
My thought about that steady increase could be because of another change in my diet. For the most of the data, I was performing bulking during my active lifestyle, and then at a later portion of time within the test data, I started to cut instead of bulking. That change in caloric consumption/macro could've been something that the model wasn't expecting, due to all the major changes in my diets, lifestyle, and fitness.
Use sequential, autoregressive prediction loop¶
One thing I wanted to try with the model, is instead of predicting the change of weight/waist based on given data, inclusive of the previous day's weight/waist measurement, that I instead make the model predict the change of weight/waist size, then on the next day, that prediction change is them added/subtracted from the previous weight/waist data. The model compounds it's prediction over and over.
This is a more realistic approach to such a model, as you wouldn't reasonably have data of every weight/waist measurement from the day before to continually feed the model in order for it to find the difference for the next day. This will show how the model attempts to predict the weight/waist on its own, based on its previous predictions rather than based on a set data table.
There are several important points to note of for this forecasting model...
Simulated Inferece Loop:¶
We have the trained model and test data, now we need a loop function, to continuously loop in the weight/waist change prediction and apply its value upon their respective column inside X_test.
Inverse standardize, then re-standardize:¶
To properly add in the prediction to the starting weight/waist, we will need to make a prediction, inverse standardize the results, as both X and y datasets use different standardization to fit to it, making both datasets incompatable unless reversed standardization. Once appended into the X test, we'll need to restandardize again for the next prediction, as that's the data the model has learnt from.
Simultaneous prediction of Weight and Waist:¶
A thing to note of, is in the X training data, it includes the weight and waist measurement. If I were to do the autoregressive prediction on weight, the same will have to be done on waist at the same time. This is because if you don't have access to future measurments of weight, then you reasonably won't have waist measurments as well. Therefore forecasting both predictions at the same time would be more realistic, than a column that gives away crucial information.
P.S. Typing this out, I realized that the test data between weight and waist is probably the same, as the only real change was their targets (y data). So I will do a boolean operation to see this in action
In [288]:
# checking if both waist and weight test data are the same or not, with 10^-13 tolerance
np.allclose(scaler_X_waist.inverse_transform(normX_waist_test), scaler_X_weight.inverse_transform(normX_weight_test), atol=1e-13)
Out[288]:
True
Yup, indeed the same model. It returns false if atol is 1e-14, which is likely due to standardization taking effect
In [ ]:
# Create sequential, autoregressive prediction loop
def simulation_loop(model_weight, model_waist, norm_X, id_waist, id_weight, X_scaler,
scaler_y_weight, scaler_y_waist, start_weight, start_waist, steps_ahead):
"""
Forecasting weight and waist values through compounding predictions
Parameters:
model_weight, model_waist: trained models
norm_X: standardized test set
id_waist, id_weight: indexed location of where waist and weight is located in X_test dataset
X_scaler, scaler_y_weight, scaler_y_waist: scalers for each respective datasets, to either fit or inverse transform when needed
start_weight, start_waist: starting value of forecast
steps_ahead: number of steps to predict/forecast
"""
current_weight = float(start_weight) # make it float, as to not get rid of decimal predictions
current_waist = float(start_waist)
predictions = [] # keep new list for all predictions
# unstandardize x_test
X_test = X_scaler.inverse_transform(norm_X)
for i in range(steps_ahead):
# prepare current index of x_test in loop to a new variable
x = X_test[i].copy()
# replace real measurements with predictions
x[id_weight] = current_weight
x[id_waist] = current_waist
# reattach the current index row to the test dataset
X_test[i] = x
# scale test for model training
norm_X = X_scaler.transform(X_test)
# predict weight and waist
weight_pred_scaled = model_weight.predict(norm_X, verbose=0)
waist_pred_scaled = model_waist.predict(norm_X, verbose=0)
# inverse scaling of y predictions (including the *100 done on data)
weight_pred = scaler_y_weight.inverse_transform(weight_pred_scaled)[i][0]/100
waist_pred = scaler_y_waist.inverse_transform(waist_pred_scaled)[i][0]/100
# update values for next loop
current_weight += weight_pred
current_waist += waist_pred
predictions.append((current_weight, current_waist))
return predictions
In [290]:
# call in function with all details needed to predict weight and waist values with this forecasting function
preds = simulation_loop(modelweight, modelwaist, normX_weight_test,
id_weight=0, id_waist=1, X_scaler=scaler_X_weight,
scaler_y_weight=scaler_y_weight, scaler_y_waist=scaler_y_waist,
start_weight=Xweight_test['Weight (kg)'].iloc[0], start_waist=Xweight_test['Waist (cm)'].iloc[0],
steps_ahead=len(normX_weight_test))
forecast_weight, forecast_waist = zip(*preds)
In [356]:
# graph out the weight and waist forecast
plt.figure(figsize=(20,5))
plt.plot(forecast_weight, label="Weight Forecast")
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label="Weight True")
plt.title("Compound Forecast Weight")
plt.legend()
plt.show()
plt.figure(figsize=(20,5))
plt.plot(forecast_waist, label="Waist Forecast")
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label="Waist True")
plt.title("Compound Forecast Waist")
plt.legend()
plt.show()
Unfortunately the compounding forecast seem to be worse performing for both the entire weight and waist data.
However, it's also worth to try the compound forecast on the active lifestyle of weight and waist as well, as we've seen how well the waist prediction has done. I'll have to create the active lifestyle weight model as well.
In [292]:
# create active lifestyle weight model
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
# Set seed for reproducibility
tf.random.set_seed(35)
# Filter training data to start post index 1150, aka where active lifestyle starts
Xweight_sequence_train_active = Xweight_sequence_train[1150:]
yweight_sequence_train_active = yweight_sequence_train[1150:]
# Ensure input shape is correct
Xweight_sequence_train_active = Xweight_sequence_train_active.reshape(Xweight_sequence_train_active.shape[0], -1)
scaler_X_weight = StandardScaler()
normX_weight_train_active = scaler_X_weight.fit_transform(Xweight_sequence_train_active)
scaler_y_weight = StandardScaler()
normY_weight_train_active = scaler_y_weight.fit_transform(yweight_sequence_train_active.reshape(-1, 1))
# Define model
modelweight_active = keras.Sequential([
layers.GaussianNoise(0.1, input_shape=(normX_weight_train.shape[1],)),
layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1)
])
lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.001,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6 # Minimum LR
)
def quantile_loss(q):
def loss(y_true, y_pred):
e = y_true - y_pred # calculate error difference between pred and truth
return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss
modelweight_active.compile(optimizer = Adam(learning_rate=lr_schedule),
loss=quantile_loss(0.7),
metrics=['mae'])
histweight_active = modelweight_active.fit(normX_weight_train_active, normY_weight_train_active,
validation_split=0.2,
epochs=1000, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [293]:
Xweight_sequence_test = Xweight_sequence_test.reshape(Xweight_sequence_test.shape[0], -1)
normX_weight_test = scaler_X_weight.transform(Xweight_sequence_test)
normY_weight_test = scaler_y_weight.transform(yweight_sequence_test.reshape(-1, 1))
show_scores(modelweight_active, normY_weight_test, normX_weight_test, 'Weight Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 27ms/step
Out[293]:
('Weight Evaluation',
{'Test MAPE (%)': np.float64(203.4087975018602),
'Test MSE': 1.7763412661267255,
'Test R^2': -0.25398206584284244})
In [294]:
# apply simulation loop function of weight
preds = simulation_loop(modelweight_active, modelwaist_active, normX_weight_test,
id_weight=0, id_waist=1, X_scaler=scaler_X_weight,
scaler_y_weight=scaler_y_weight, scaler_y_waist=scaler_y_waist,
start_weight=Xweight_test['Weight (kg)'].iloc[0], start_waist=Xweight_test['Waist (cm)'].iloc[0],
steps_ahead=len(normX_weight_test))
forecast_weight_active, forecast_waist_active = zip(*preds)
In [295]:
# visualize graph of active lifestyle models
plt.figure(figsize=(20,5))
plt.plot(forecast_weight_active, label="Weight Active Lifestyle Forecast")
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label="Weight True")
plt.title("Weight Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()
plt.figure(figsize=(20,5))
plt.plot(forecast_waist_active, label="Waist Active Lifestyle Forecast")
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label="Waist True")
plt.title("Waist Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()
It does seem the forecast prediction on active lifestyle models, give better results. However it still suffers from the same problem of over estimating values. Though for some reson, the weight active lifestyle model is the only one underperforming from the rest and is the most accurate overall.
My thoughts as to weight model, is likely because I didn't use the weight model to outfit the active lifestyle weight model. Instead, I copy pasted the waist active lifestyle over to weight, and changed the variable name.
I know for a fact that waist and active waist model were the same, however I did tweaking with the weight model instead. It seems clear that I've messed up the model for weight, and that both waist/weight data are similar enough to be able to exchange the same model, and train it separately from each other.
Trying new Quartile Loss¶
As I was finishing up my report pdf for this project, I decided to do further research on what quartile loss is and what it does. ChatGPT gave me the impression that the higher the quartile loss is, the more likely the model makes riskier predictions.
Then seeing a graph output from google image, meant that it's predicting at certain quartile levels of the y data. Aka, the higher the quartile, the more likely the model predicts at the higher bound of the range of y data. ChatGPT was technically right with stating how larger q loss makes for riskier prediction, whilst more general prediction at lower, as the predictions hovered more closer to 0.
This would then explain why the model continuously underpredicts the negative values. As the quartile loss on all my model was 0.7, with the exception of weight model (the entire dataset version), where it was 0.9. And it was evident that weight model prediction has the highest overestimation among the models.
So I'll be experimenting with quartile loss on the active models, since they make the best predictions.
In [325]:
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
# Set seed for reproducibility
tf.random.set_seed(35)
# Filter training data to start post index 1150, aka where active lifestyle starts
Xwaist_sequence_train_active = Xwaist_sequence_train[1150:]
ywaist_sequence_train_active = ywaist_sequence_train[1150:]
# Ensure input shape is correct
Xwaist_sequence_train_active = Xwaist_sequence_train_active.reshape(Xwaist_sequence_train_active.shape[0], -1)
scaler_X_waist = StandardScaler()
normX_waist_train_active = scaler_X_waist.fit_transform(Xwaist_sequence_train_active)
scaler_y_waist = StandardScaler()
normY_waist_train_active = scaler_y_waist.fit_transform(ywaist_sequence_train_active.reshape(-1, 1))
# Define model
modelwaist_active = keras.Sequential([
layers.GaussianNoise(0.1, input_shape=(normX_waist_train.shape[1],)),
layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1)
])
lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.001,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6 # Minimum LR
)
def quantile_loss(q):
def loss(y_true, y_pred):
e = y_true - y_pred # calculate error difference between pred and truth
return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss
modelwaist_active.compile(optimizer = Adam(learning_rate=lr_schedule),
loss=quantile_loss(0.7),
metrics=['mae'])
histwaist_active = modelwaist_active.fit(normX_waist_train_active, normY_waist_train_active,
validation_split=0.2,
epochs=1000, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [307]:
# create active lifestyle weight model
import tensorflow as tf
from tensorflow import keras
from sklearn.preprocessing import StandardScaler
from tensorflow.keras import backend as K
# Set seed for reproducibility
tf.random.set_seed(35)
# Filter training data to start post index 1150, aka where active lifestyle starts
Xweight_sequence_train_active = Xweight_sequence_train[1150:]
yweight_sequence_train_active = yweight_sequence_train[1150:]
# Ensure input shape is correct
Xweight_sequence_train_active = Xweight_sequence_train_active.reshape(Xweight_sequence_train_active.shape[0], -1)
scaler_X_weight = StandardScaler()
normX_weight_train_active = scaler_X_weight.fit_transform(Xweight_sequence_train_active)
scaler_y_weight = StandardScaler()
normY_weight_train_active = scaler_y_weight.fit_transform(yweight_sequence_train_active.reshape(-1, 1))
# Define model
modelweight_active = keras.Sequential([
layers.GaussianNoise(0.1, input_shape=(normX_weight_train.shape[1],)),
layers.Dense(512, kernel_regularizer=regularizers.l2(0.001)),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(256),
layers.BatchNormalization(),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(128),
layers.LeakyReLU(alpha=0.1),
layers.Dropout(0.1),
layers.Dense(64),
layers.LeakyReLU(alpha=0.1),
layers.Dense(1)
])
lr_schedule = tf.keras.optimizers.schedules.CosineDecayRestarts(
initial_learning_rate=0.001,
first_decay_steps=200, # Will start decaying learning rate, after 200 epochs
t_mul=2.0, # Reduces decay overtime, multiplying by 2 on the first decay steps (200*2 = 400 is the new epoch to decay)
m_mul=0.8, # Reduces peak LR over time, where our start is 0.001, multiply by 0.8, gives us 0.0008 for new start of lr
alpha=1e-6 # Minimum LR
)
def quantile_loss(q):
def loss(y_true, y_pred):
e = y_true - y_pred # calculate error difference between pred and truth
return K.mean(K.maximum(q * e, (q - 1) * e)) # formula to return the loss, deterministic by q value
return loss
modelweight_active.compile(optimizer = Adam(learning_rate=lr_schedule),
loss=quantile_loss(0.65),
metrics=['mae'])
histweight_active = modelweight_active.fit(normX_weight_train_active, normY_weight_train_active,
validation_split=0.2,
epochs=1000, verbose=0)
C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\regularization\gaussian_noise.py:29: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead. super().__init__(**kwargs) C:\Users\deanc\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\keras\src\layers\activations\leaky_relu.py:41: UserWarning: Argument `alpha` is deprecated. Use `negative_slope` instead. warnings.warn(
In [326]:
# apply simulation loop function of weight
preds = simulation_loop(modelweight_active, modelwaist_active, normX_weight_test,
id_weight=0, id_waist=1, X_scaler=scaler_X_weight,
scaler_y_weight=scaler_y_weight, scaler_y_waist=scaler_y_waist,
start_weight=Xweight_test['Weight (kg)'].iloc[0], start_waist=Xweight_test['Waist (cm)'].iloc[0],
steps_ahead=len(normX_weight_test))
forecast_weight_active, forecast_waist_active = zip(*preds)
In [357]:
# visualize graph of active lifestyle models
plt.figure(figsize=(15,5))
plt.plot(forecast_weight_active, label="Weight Active Lifestyle Forecast")
plt.plot(range(len(Xweight_test)), Xweight_test['Weight (kg)'], label="Weight True")
plt.title("Weight Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()
plt.figure(figsize=(15,5))
plt.plot(forecast_waist_active, label="Waist Active Lifestyle Forecast")
plt.plot(range(len(Xwaist_test)), Xwaist_test['Waist (cm)'], label="Waist True")
plt.title("Waist Compound Forecast of Active Lifestyle")
plt.legend()
plt.show()
Report presentations:¶
These are just some extras to include inside the report
In [328]:
Xweight_sequence_test = Xweight_sequence_test.reshape(Xweight_sequence_test.shape[0], -1)
normX_weight_test = scaler_X_weight.transform(Xweight_sequence_test)
normY_weight_test = scaler_y_weight.transform(yweight_sequence_test.reshape(-1, 1))
show_scores(modelweight_active, normY_weight_test, normX_weight_test, 'Weight Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 6ms/step
Out[328]:
('Weight Evaluation',
{'Test MAPE (%)': np.float64(206.4920772698088),
'Test MSE': 1.77812658512935,
'Test R^2': -0.2552423856101027})
In [329]:
Xwaist_sequence_test = Xwaist_sequence_test.reshape(Xwaist_sequence_test.shape[0], -1)
normX_waist_test = scaler_X_waist.transform(Xwaist_sequence_test)
normY_waist_test = scaler_y_waist.transform(ywaist_sequence_test.reshape(-1, 1))
show_scores(modelwaist_active, normY_waist_test, normX_waist_test, 'Waist Evaluation')
5/5 ━━━━━━━━━━━━━━━━━━━━ 0s 7ms/step
Out[329]:
('Waist Evaluation',
{'Test MAPE (%)': np.float64(224.53574301705336),
'Test MSE': 1.602273135015225,
'Test R^2': -0.18429371657718363})
In [350]:
plt.plot(histwaist_active.history['loss'])
plt.xlabel('Epoch')
plt.ylabel('Loss')
plt.title('Loss over Epoch')
Out[350]:
Text(0.5, 1.0, 'Loss over Epoch')
In [351]:
import numpy as np
# function for monte carlo dropout inference
def monte_carlo_predictions(model, X, n_simulation=100):
preds = np.array([model(X, training=True).numpy().flatten() for _ in range(n_simulation)])
return preds.mean(axis=0), preds.std(axis=0)
mean_pred_weight_active, uncertainty_active_weight = monte_carlo_predictions(modelweight_active, normX_weight_test, n_simulation=100)
In [353]:
import matplotlib.pyplot as plt
plt.figure(figsize=(15, 5))
plt.plot(range(len(mean_pred_weight_active)), mean_pred_weight_active, label="Waist Prediction", color='blue')
plt.plot(normY_weight_test, label='True waist', color='red')
plt.fill_between(range(len(mean_pred_weight_active)), mean_pred_weight_active - 2 * uncertainty_active_weight, mean_pred_weight_active + 2 * uncertainty_active_weight,
color='blue', alpha=0.2, label="Confidence Interval (±2σ)")
plt.xlabel("Sample Index")
plt.ylabel("Prediction")
plt.title("Model Prediction (with Standard Deviation) and Truth")
plt.legend()
plt.show()
In [345]:
fig, ax1 = plt.subplots(figsize=(20,5))
ax1.set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1.plot(df['Date'][1150+61:1150+61+306], df['Daily intake (kcal)'][1150+61:1150+61+306], label='Calorie consumption in kcal', color='orange', linewidth=1) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1.twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(df['Date'][1150+61:1150+61+306], df['Weight (kg)'][1150+61:1150+61+306], label='Weight in kg', linewidth=1)
ax1.legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend()
Out[345]:
<matplotlib.legend.Legend at 0x1af03dfb710>
In [349]:
fig, ax1 = plt.subplots(figsize=(20,5))
ax1.set(title='Weight and Calorie consumption overtime', xlabel='Date')
ax1.plot(df['Date'][700:700+306], df['Daily intake (kcal)'][700:700+306], label='Calorie consumption in kcal', color='orange', linewidth=1) # separating this data point with a specific colour, but also shrinking line width to make it readable
ax2 = ax1.twinx() # creating secondary y axis, and it's associated axis name
ax2.plot(df['Date'][700:700+306], df['Weight (kg)'][700:700+306], label='Weight in kg', linewidth=1)
ax1.legend(loc='lower center') # putting legend at lower center, where it won't overlap the data
ax2.legend()
Out[349]:
<matplotlib.legend.Legend at 0x1af05731c90>
In [358]:
pip install nbconvert PyPDF2
Requirement already satisfied: nbconvert in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (7.16.6) Requirement already satisfied: PyPDF2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (3.0.1) Requirement already satisfied: beautifulsoup4 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (4.13.3) Requirement already satisfied: bleach!=5.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from bleach[css]!=5.0.0->nbconvert) (6.2.0) Requirement already satisfied: defusedxml in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (0.7.1) Requirement already satisfied: jinja2>=3.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (3.1.5) Requirement already satisfied: jupyter-core>=4.7 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (5.7.2) Requirement already satisfied: jupyterlab-pygments in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (0.3.0) Requirement already satisfied: markupsafe>=2.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (3.0.2) Requirement already satisfied: mistune<4,>=2.0.3 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (3.1.1) Requirement already satisfied: nbclient>=0.5.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (0.10.2) Requirement already satisfied: nbformat>=5.7 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (5.10.4) Requirement already satisfied: packaging in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (24.2) Requirement already satisfied: pandocfilters>=1.4.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (1.5.1) Requirement already satisfied: pygments>=2.4.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (2.19.1) Requirement already satisfied: traitlets>=5.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbconvert) (5.14.3) Requirement already satisfied: webencodings in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from bleach!=5.0.0->bleach[css]!=5.0.0->nbconvert) (0.5.1) Requirement already satisfied: tinycss2<1.5,>=1.1.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from bleach[css]!=5.0.0->nbconvert) (1.4.0) Requirement already satisfied: platformdirs>=2.5 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-core>=4.7->nbconvert) (4.3.6) Requirement already satisfied: pywin32>=300 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-core>=4.7->nbconvert) (308) Requirement already satisfied: jupyter-client>=6.1.12 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbclient>=0.5.0->nbconvert) (8.6.3) Requirement already satisfied: fastjsonschema>=2.15 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbformat>=5.7->nbconvert) (2.21.1) Requirement already satisfied: jsonschema>=2.6 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from nbformat>=5.7->nbconvert) (4.23.0) Requirement already satisfied: soupsieve>1.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from beautifulsoup4->nbconvert) (2.6) Requirement already satisfied: typing-extensions>=4.0.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from beautifulsoup4->nbconvert) (4.12.2) Requirement already satisfied: attrs>=22.2.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (25.1.0) Requirement already satisfied: jsonschema-specifications>=2023.03.6 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (2024.10.1) Requirement already satisfied: referencing>=0.28.4 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (0.36.2) Requirement already satisfied: rpds-py>=0.7.1 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jsonschema>=2.6->nbformat>=5.7->nbconvert) (0.22.3) Requirement already satisfied: python-dateutil>=2.8.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (2.9.0.post0) Requirement already satisfied: pyzmq>=23.0 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (26.2.1) Requirement already satisfied: tornado>=6.2 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (6.4.2) Requirement already satisfied: six>=1.5 in c:\users\deanc\appdata\local\packages\pythonsoftwarefoundation.python.3.11_qbz5n2kfra8p0\localcache\local-packages\python311\site-packages (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->nbclient>=0.5.0->nbconvert) (1.17.0) Note: you may need to restart the kernel to use updated packages.
[notice] A new release of pip is available: 24.0 -> 25.0.1 [notice] To update, run: C:\Users\deanc\AppData\Local\Microsoft\WindowsApps\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\python.exe -m pip install --upgrade pip
In [ ]: